Intentionally left empty @DominikDataDev

Handling Schema Evolution in the Data Pipelines at KOHO by @shwetastha1O koho.dev/handling-schem… One possible way to automatically handling source schema changes when loading data into AWS Redshift

0 0 0 13 0

Intentionally left empty @DominikDataDev

2 years ago

Open AI is nothing without its people

0 0 0 10 0

Intentionally left empty @DominikDataDev

2 years ago

„You can’t unit test SQL“ Well, that´s wrong. And this should be obvious because queries are conceptually so similar to dataframe transformations. The simple trick is to limit the scope of CTEs as if they were functions. Example without 3rd party tools: stackoverflow.com/a/754570

0 0 0 13 0

Intentionally left empty @DominikDataDev

2 years ago

15 Essential Steps To Build Reliable Data Pipelines towardsdatascience.com/15-essential-s…

0 0 0 8 0

Intentionally left empty @DominikDataDev

2 years ago

10 Reasons Why Estimating Time For Data Projects is Hard annageller.com/blog/10-reason…

0 0 0 7 0

Joseph Machado @startdataeng

3 years ago

When data to process is larger than memory, try to stream with python generators, before jumping to distributed systems! #data #dataengineering #Python #Pythonlearning #Generator E.g. Stream a file(note () and not []), get diff between date cols

6 77 387 0 268

Download Image

Intentionally left empty @DominikDataDev

2 years ago

Good summary on how to structure Pyspark code for Unit testing, including approaches to test data engineeringfordatascience.com/posts/pyspark_…

0 0 0 17 0

kache @yacineMTB

2 years ago

the best engineer I've ever met in my life told me something that's always stuck with me: "software is only correct given a certain time frame" the world changes, and your software needs to change along with it