Joseph Machado @startdataeng

I write about data engineering | SQL | Python | Distributed systems. Get my free data engineering course at https://t.co/sZTEcV0Q9W startdataengineering.com New york Joined April 2020

Tweets

2K
Followers

9K
Following

38
Likes

548

Joseph Machado @startdataeng

4 years ago

It can be overwhelming to start learning data engineering. I'd recommend starting with the basics of python, sql, UNIX commands, building a simple data project, update Github, Linkedin. Landing a DE job is 60% part learning and 40% marketing. See reply 👇🏽 for helpful links.

15 80 358 0 440

Joseph Machado @startdataeng

3 years ago

Starting as a DE? 90% of what you will need is SQL (OLAP), python, & distributed system basics Don't overcomplicate! #data #dataengineering #SQL #database #Python

21 73 459 0 230

Joseph Machado @startdataeng

5 years ago

Exercise project for anyone starting in data engineering startdataengineering.com/post/data-engi… #dataengineering #bigdata #ETL #ApacheAirflow #AWS #ApacheSpark

12 93 482 0 426

Download Image

Joseph Machado @startdataeng

3 years ago

Preparing for SQL interviews? Do Leetcode SQL hard, sort by freq, and do the first 40 #data #dataengineering #Software #SQL

14 73 381 0 290

Joseph Machado @startdataeng

3 years ago

Backfilling is an inevitable part of data projects. When designing your data pipelines take some time to answer the following questions 1. Does multiple backfill runs cause duplicate data? 2. Can multiple backfills be parallelized? #data #dataEngineering #datapipeline #datasets

4 23 204 0 98

Joseph Machado @startdataeng

3 years ago

Learning data engineering? Build a pipeline locally. 1. Python to pull data from an API (e.g. Coincap) 2. Load data into a local Postgres container 3. Automate it with cron/task scheduler Start small, build, improve, & repeat. #data #dataengineering #Pythonlearning #Python

9 108 523 0 415

Joseph Machado @startdataeng

3 years ago

Left anti-join is cool! Get all the data from the left table that has no matching data in the right table select t1.* from t1 left join t2 on t1.id=t2.id where t2.id is null; #data #dataengineering #SQL #database

6 18 138 0 46

Joseph Machado @startdataeng

3 years ago

Starting a data project is a lot of work! It can be overwhelming to define the problem, set up systems, and then code! Use this DE project as a blueprint to build your own: startdataengineering.com/post/data-engi… #data #dataengineering #database #dataAnalytics #dataviz #Python #datapipeline

5 77 365 0 386

Download Image

Joseph Machado @startdataeng

5 years ago

If you are interested in using "Change Data Capture" pattern for streaming ETL, check out startdataengineering.com/post/change-da… #ETL #changedatacapture #dataengineering #debezium #BigData

5 64 287 0 210

Download Image

Joseph Machado @startdataeng

3 years ago

When data to process is larger than memory, try to stream with python generators, before jumping to distributed systems! #data #dataengineering #Python #Pythonlearning #Generator E.g. Stream a file(note () and not []), get diff between date cols

6 77 387 0 268

Download Image

Joseph Machado @startdataeng

3 years ago

An orchestration tool that I've been impressed with is @dagsterio. Easy setup, powerful features and great docs. Use 👇🏽 to play around with a pipeline on dagster startdataengineering.com/post/data-engi… #data #dataengineering #Python #database #dataAnalytics

4 71 290 0 223

Joseph Machado @startdataeng

7 months ago

uv by @astral_sh is truly one of the best tools you can have in your toolkit as a DE. TIL: You can quickly start a jupyter notebook with it doc: docs.astral.sh/uv/guides/inte…

0 0 2 382 2

Joseph Machado @startdataeng

7 months ago

Data engineers write the most complex piece of code to Upsert into tables. Here's THE command you need to know MERGE INTO/INSERT ON CONFLICT #data #dataengineering #SQL