Picture

Bartosz Konieczny @waitingforcode

Freelance Data Engineer and instructor, enjoy solving data problems with #ApacheSpark #AWS #GCP #Azure 👨‍🏭 | [email protected] waitingforcode.com remote Joined January 2011
  • Tweets
    857
  • Followers
    2K
  • Following
    79
  • Likes
    749
Bartosz Konieczny retweeted
shroffpub Profile Picture

Shroff Publishers @shroffpub

4 months ago

Releasing Soon! Pre-order now shroffpublishers.com/books/97893680… Data Engineering Design Patterns By Bartosz Konieczny  @waitingforcode. with @OReillyMedia Focusing on various aspects of data engineering, including data ingestion, data quality, idempotency, and more. #dataengineering

Bartosz Konieczny retweeted
vanlightly Profile Picture

Jack Vanlightly @vanlightly

11 months ago

If you want to understand the consistency models of the mentioned table formats of the paper, I've written about it extensively and written formal models. * jack-vanlightly.com/analyses/2024/… * jack-vanlightly.com/analyses/2024/… * jack-vanlightly.com/analyses/2024/… * github.com/Vanlightly/tab…

Bartosz Konieczny retweeted
leanpub Profile Picture

Leanpub @leanpub

a year ago

Data Engineering patterns on the cloud by Bartosz Konieczny is on sale on Leanpub! Its suggested price is $39.00; get it for $24.65 with this coupon: leanpub.com/sh/ygsnqbRD @waitingforcode #CloudComputing #AmazonWebServices #GoogleCloudPlatform #MicrosoftAzure

Bartosz Konieczny retweeted
DeltaLakeOSS Profile Picture

Delta Lake @DeltaLakeOSS

2 years ago

Join @newfront and @waitingforcode and learn all about streaming Delta Lake tables with Apache Spark Structured Streaming! 🦀 🗓 March 21st 🕝 9:00AM PT / 12:00PM ET 💻 Join this webinar via LinkedIn, YouTube, or Zoom! Learn more: linkedin.com/events/streami… #deltalake #streaming

Bartosz Konieczny retweeted
jim_dowling Profile Picture

Jim Dowling @jim_dowling

2 years ago

I have been busy the last few months writing a book for O'Reilly about how to build ML systems (batch, real-time, and LLMs), distilling much of what I have learnt from both working with customers as well as students. Why could the book interest you? * Data Scientists - transition…

Bartosz Konieczny retweeted
gwenshap Profile Picture

Gwen (Chen) Shapira @gwenshap

2 years ago

I don't want to start a flame war here, but IMO it is a mistake to jump straight to distributed databases (and 90% of the content below is distributed databases) without first learning fundamentals on single node databases. Here's my 10 things to understand about databases:…

thegeeknarrator Profile Picture

Kaivalya Apte - The Geek Narrator @thegeeknarrator

2 years ago

I don't want to start a flame war here, but IMO it is a mistake to jump straight to distributed databases (and 90% of the content below is distributed databases) without first learning fundamentals on single node databases. Here's my 10 things to understand about databases:…

Bartosz Konieczny retweeted
leanpub Profile Picture

Leanpub @leanpub

2 years ago

Data Engineering patterns on the cloud by Bartosz Konieczny is on sale on Leanpub! Its suggested price is $39.00; get it for $26.10 with this coupon: leanpub.com/sh/1T4q5Z81 @waitingforcode #CloudComputing #AmazonWebServices #GoogleCloudPlatform #MicrosoftAzure

Bartosz Konieczny retweeted
vanlightly Profile Picture

Jack Vanlightly @vanlightly

2 years ago

Chapter 4 of The Architecture of Serverless Data Systems: CockroachDB (serverless). jack-vanlightly.com/analyses/2023/…

Bartosz Konieczny retweeted
DeltaLakeOSS Profile Picture

Delta Lake @DeltaLakeOSS

2 years ago

The early release of Delta Lake: The Definitive Guide is here! 🎉 The latest edition includes the addition of Chapter 12: Performance Tuning. Download here ➡️ bit.ly/472DVY7 Authors @dennylee, Prashanth Babu, Tristen Wentling, & @newfront #opensource #deltalake #oss

Bartosz Konieczny retweeted
leanpub Profile Picture

Leanpub @leanpub

2 years ago

Data Engineering patterns on the cloud: How to solve common data engineering problems with cloud services? leanpub.com/data-engineeri… by Bartosz Konieczny is the featured book on the Leanpub homepage! leanpub.com @waitingforcode #CloudComputing #AmazonWebServices

waitingforcode Profile Picture

Bartosz Konieczny @waitingforcode

2 years ago

Last week I spent some time to understand the #PySpark applyInPandasWithState. This week I'm refactoring the code, hoping to still understand it 2 months later ;) 👉 waitingforcode.com/apache-spark-s…

waitingforcode Profile Picture

Bartosz Konieczny @waitingforcode

2 years ago

In the previous release #PySpark has got an interesting streaming feature -> the arbitrary stateful processing. It has a different API than the Scala version but is more adapted to the Python world. More 👉 waitingforcode.com/apache-spark-s…

Bartosz Konieczny retweeted
antonmry Profile Picture

Antón @antonmry

2 years ago

A list of articles I share again and again when developers ask me about Kafka 🧵

Bartosz Konieczny retweeted
ApacheSpark Profile Picture

Apache Spark @ApacheSpark

2 years ago

[ANNOUNCEMENT] Congrats to the Apache Spark community and all the contributors! The Apache Spark 3.5.0 release is here. Try it out! spark.apache.org/releases/spark…

waitingforcode Profile Picture

Bartosz Konieczny @waitingforcode

2 years ago

It's not a rebranding but more a regrouping 😉 All my additional #dataengineering content is now available from there waitingforcode.com/better (planning to add some stream processing materials soon)

waitingforcode Profile Picture

Bartosz Konieczny @waitingforcode

2 years ago

If Delta Lake implemented the commits only, I could stop exploring this transactional part after the previous article. But as for RDBMS, #DeltaLake implements other ACID-related concepts, such as isolation levels 👉 waitingforcode.com/delta-lake/tab…

waitingforcode Profile Picture

Bartosz Konieczny @waitingforcode

2 years ago

One of the great features of table file formats is the ability to handle write conflicts. It wouldn't be possible without commits that are the topic of my #DeltaLake blog post. waitingforcode.com/delta-lake/tab…

waitingforcode Profile Picture

Bartosz Konieczny @waitingforcode

2 years ago

Surprises may be hidden elsewhere, even in the provider-managed libraries. I got punished once for relying on them without verifying the ins and outs before. Lessons learned 👉 waitingforcode.com/data-engineeri…

waitingforcode Profile Picture

Bartosz Konieczny @waitingforcode

2 years ago

OOM problems in #ApacheSpark Structured Streaming were often due to the infinitely growing metadata layer. There were a few workarounds but it's also possible to use a proper configuration, at least for file sink 👉 waitingforcode.com/apache-spark-s…

waitingforcode Profile Picture

Bartosz Konieczny @waitingforcode

2 years ago

If you rely on the watermark for the state expiration in #ApacheSpark arbitrary stateful processing, be careful. The first micro-batch doesn't contain the watermark yet! You can find some of possible workarounds in the new blog post 👉 waitingforcode.com/apache-spark-s…

Trends for United States

Chiefs
194 B posts
Rachel
68,7 B posts
Herbert
54,5 B posts
Karen
73,6 B posts
Mahomes
66,1 B posts
AJ Lee
229 B posts
#SmackDown
185 B posts
Kelce
40,8 B posts
Worthy
54,8 B posts
#KCvsLAC
7.855 posts
Keenan Allen
5.346 posts
Big Brother
23,8 B posts
Rich Eisen
5.417 posts
#BoltUp
6.960 posts
Ken Dryden
2.636 posts
Chris Jones
5.135 posts
Brazil
124 B posts
Quentin Johnston
5.889 posts
Jawaan Taylor
2.661 posts
Madison Beer
3.973 posts

You might like

Lightning-fast unified analytics engine
Apache Spark
@ApacheSpark
42K Followers
Cofounder @Databricks
Reynold Xin
@rxin
13K Followers
#DataAISummit (formerly #SparkAISummit) is the global event for the data community. The conference is organized by @Databricks.
#DataAISummit
@Data_AI_Summit
20K Followers
Official twitter handle of Apache Hudi, an open data lakehouse platform. https://t.co/SXay7oHNah
Apache Hudi
@apachehudi
3K Followers
Freelance Data(bricks) Engineer • #ApacheSpark #DeltaLake #UnityCatalog #MLflow #Databricks #DSPy | Java Champion | @theASF | #DatabricksMVP
Jacek Laskowski
@jaceklaskowski
7K Followers
Modern, open source data exploration & visualization platform 📊. Open Source business intelligence (BI) is here, and here to win!
Apache Superset
@apachesuperset
5K Followers
CTO at @Databricks and CS prof at @UCBerkeley. Working on data+AI, including @ApacheSpark, @DeltaLakeOSS, @MLflow, https://t.co/94gROE5Xa0. https://t.co/nmRYAKG0LZ
Matei Zaharia
@matei_zaharia
44K Followers
Director @ Confluent • Author • Databricks MVP • From distributed data to real-world AI systems
Adi Polak
@AdiPolak
15K Followers
I write about data engineering | SQL | Python | Distributed systems. Get my free data engineering course at https://t.co/sZTEcV0Q9W
Joseph Machado
@startdataeng
9K Followers
An open source machine learning platform for managing the complete ML lifecycle
MLflow
@MLflow
10K Followers
Real-time distributed OLAP datastore. Point at Kafka and start querying. Join our growing community at https://t.co/s0Tg6jKf8W
Apache Pinot
@ApachePinot
5K Followers
Delta Lake is an open-source storage framework that enables building a Lakehouse architecture for Spark, Flink, Trino, Hive, Scala, Java, Rust, Python, & more!
Delta Lake
@DeltaLakeOSS
10K Followers
Flink Forward is the conference dedicated to @ApacheFlink and the stream processing community.
Flink Forward
@FlinkForward
3K Followers
A distributed streaming platform. Account managed by the Kafka PMC.
Apache Kafka
@apachekafka
68K Followers
Founder @Onehousehq, Creator of @apachehudi, Built the World's first #DataLakehouse, Distributed/Data Systems, Linkedin, Uber, Confluent alum. (views are mine)
Vinoth Chandar
@byte_array
2K Followers