Querying Billions of GitHub Events Using Modal and DuckDB
noreasontopanic.com/p/querying-bil…
Very cool demo of how to use Modal for fan-out parallelism
did some tuning and can now download the last 5 years of public @github events in ~15m
3.5 TB of data (50k+ json.gz files) big fanout made super easy using @modal 🎉, lots of parallel downloads + retries with exponential back-off
blog post and open-source repo coming soon!
did some tuning and can now download the last 5 years of public @github events in ~15m
3.5 TB of data (50k+ json.gz files) big fanout made super easy using @modal 🎉, lots of parallel downloads + retries with exponential back-off
blog post and open-source repo coming soon! https://t.co/ZMnz44uxK5
PSA if you haven't used python in the past 5 years it's a completely new language now
- uv solved the package management problems entirely
- mypy type checks have become actually useful
- cpython is now much faster
- polars + fastapi + ruff replaced pandas + flask + black
I'm rediscovering python after a few years, and between @astral_sh and @modal_labs, I'm completely blown away by how fun it is.
no more env/dep insanity, and practically instant remote code execution 🎉
7K Followers 2K FollowingIt’s like Zack with an R. Bay native 🇺🇸🇮🇳. Product lad turned partner @BainCapVC. Investing in AI, developers, infra, & security. Grew up @Atlassian
241 Followers 477 Following||OAU|| Ogun | Student Baker | Scrabble ❤️ | NASELS | Official Campus Ambassador at International MUN @imunofficial ✨🇺🇳 || Helping the growth @WayoWork
6K Followers 7K FollowingFOLLOWS YOU 🫵 https://t.co/F7MzDOTC1k
ML/AI, R&D eng, quant trading, ASR in noise, TTS.
OPEN weights, thoughts, ... AGI, ASI - open AI computation for */acc—NOW 🥰
14K Followers 15K FollowingAustin Powered. Co-founder of OpenStack & OpenInfra Foundation. General Manager of AI & Infrastructure for the Linux Foundation. open source for fun & profit.
14K Followers 1K Following“All methods are sacred if they are internally necessary” (GP @amplifypartners, prev @canvasvc; Head of Data @Mattermark; @palantirtech; @c4ads)
416K Followers 2K Following@Shopify CEO by day, Dad in evening, hacker at night. Aspiring comprehensivist. (tweets auto delete eventually) retweet=noteworthy share, not endorsement
10K Followers 2K FollowingCartoonist, Engineer, PM, Partner @a16z investing in infra & AI
Prev Product lead @HashiCorp, Founding Eng/PM @Transposit. Eng @AppDynamics. Opinions = own.
13K Followers 5K FollowingLorin Hochstein
Student of complex systems failures, resilience eng, cognitive sys eng. Will talk your ear off about @LFISoftware.
@[email protected]
8K Followers 157 FollowingMaking analytics fun, frictionless and ducking awesome with an unorthodox data warehouse for the post big data era based on @DuckDB in collab with @duckdblabs.
7K Followers 1K Following👾👾 Co-Founder of @RunReveal . 🐄🐄 Former Cloudflare Sr. Director of Security Engineering. JMU Double Duke. There's always more to learn.
15K Followers 2K FollowingFounder @RillData, the fastest path to operational BI. Previously founded @Metamarkets, @DCVC, @CustomInk. Lapsed computational biologist.
20K Followers 2 FollowingDuckDB is an analytical in-process SQL database management system. "DuckDB" and the DuckDB logo are registered trademarks of the DuckDB Foundation.
17K Followers 20 FollowingA high-throughput and memory-efficient inference and serving engine for LLMs. Join https://t.co/lxJ0SfX5pJ to discuss together with the community!
8K Followers 5K Followingceo @tembo (the ai coding automation platform) · founder/ex-ceo @astronomerio (yes, that astronomer) · I follow back all devs, dm if i missed you
4K Followers 1K FollowingBoris Babayan said "architecture, operating system and languages, compiler, it's only one project". Investing in improvements to the project @AmplifyPartners.
2K Followers 69 FollowingIndiehacker. #BullMQ - https://t.co/uzk4uaxk1r - https://t.co/ECOwuvxKA3.
Follow my journey in trying to build profitable businesses around Open Source.
1.7M Followers 91 FollowingOfficial account of Ray Dalio, founder of Bridgewater Associates, author of #1 New York Times bestseller 'Principles,' professional mistake maker
269K Followers 5K FollowingPosts about Business. Building SaaS for 20+ years. Started https://t.co/xrOkg6UWaz (2005), KISSmetrics (2008) & https://t.co/uA827W8uqv (2020). What are you building? 🙏
39K Followers 2K FollowingI cover SCOTUS @TheEconomist and teach @BHSEC. Brooklynite, @UMich pol science PhD, dad of three, runner. Views here are mine.
41K Followers 940 FollowingAuthor of Hannah Arendt and What Remains.
Associate Faculty @BklynInstitute
Writing a book about loneliness.
Substack: https://t.co/S2b4vQt7U3