llm-d @_llm_d_
llm-d: a Kubernetes-native high-performance distributed LLM inference framework llm-d.ai Joined May 2025-
Tweets38
-
Followers282
-
Following2
-
Likes3
Austin friends - we’ve got something special coming up on Wednesday, September 17 at @CapitalFactory! PyTorch ATX + vLLM are teaming up for a joint meetup. Over 200 folks are already signed up, and there’s still a few spots left. Come hang out, learn, and meet others building…
Let's break down intelligent inference serving. Traditional serving uses basic round-robin load balancing where requests are sent to the "next" pod. Intelligent inference serving makes scheduling decisions based on AI-specific workload signals. Let's dig into what this means.
Traditional load balancing fails with stateful LLMs. That's why the llm-d contributors built an AI-aware scheduler. The result? 100% request success & massive performance gains in Kubernetes. 🚀 🔗 Read the latest post: llm-d.ai/blog/intellige…
We're excited to be on the latest Kubernetes Podcast! Clayton Coleman (@Google) & Rob Shaw (@RedHat_AI ) discuss how llm-d is building a well-lit path for distributed LLM inference and why AI workloads are a unique challenge for Kubernetes Listen here: kubernetespodcast.com/episode/258-ll…
@KubeCon_ We'll dive into the latest challenges in serving LLMs and showcase the integrations in KServe with @_llm_d_, @vllm_project, @lmcache, Envoy, and more! @CloudNativeFdn @kubernetesio @RedHat_AI #KubeCon #CloudNativeCon #MLOps #AI #DevOps #Kubernetes #K8s #CloudNative #CNK8sAIDay
Excited to share that I'll be speaking at 𝗖𝗹𝗼𝘂𝗱 𝗡𝗮𝘁𝗶𝘃𝗲 𝗞𝟴𝘀 𝗔𝗜 𝗗𝗮𝘆, in addition to @KubeCon_! Dan Sun and I will be delivering 𝗞𝗦𝗲𝗿𝘃𝗲 𝗡𝗲𝘅𝘁: 𝗔𝗱𝘃𝗮𝗻𝗰𝗶𝗻𝗴 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗔𝗜 𝗠𝗼𝗱𝗲𝗹 𝗦𝗲𝗿𝘃𝗶𝗻𝗴: sched.co/28D4J
🚀 𝗣𝗿𝗼𝗴𝗿𝗲𝘀𝘀 𝗨𝗽𝗱𝗮𝘁𝗲: 𝗹𝗹𝗺-𝗱 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗞𝗦𝗲𝗿𝘃𝗲 🚀 Scaling LLM serving brings challenges far beyond a single deployment or @kubernetesio’ default autoscaling.
Huge thanks to Jeff Fan & Anish Singh Walia from @digitalocean for writing this amazing deep-dive on deploying llm-d on DOKS! 🎉 A fantastic community contribution It's a great, hands-on guide for anyone looking to run llm-d on DOKS. digitalocean.com/community/tuto… #LLM #Kubernetes
Here's the agenda: 💡 Intro to vLLM for Fast and Efficient AI Inference 📈 High-Performance LLM Scaling with @_llm_d_ ⚙️ Deep dive into vLLM TPU integration 🏗️ Building and Testing Infrastructure for vLLM ⚡️ Supercharging @Rebellions_inc NPUs with vLLM + Networking and Q&A.
Getting started with llm-d v0.2 is now easier than ever! We've launched a full set of quick start guides to walk you through our most powerful features, including P/D disaggregation and deploying large MoE models on Kubernetes. Start here: llm-d.ai/docs/guide
Awesome lineup of speakers today! See you there?!
Awesome lineup of speakers today! See you there?! https://t.co/4j7qKTMTyS
📣 TODAY at 2pm EST 📣 Want to learn how to scale MoE models like DeepSeek R1 on llm-d? Join the virtual vLLM Office hours TODAY at 2pm EST and learn to scale these models with high performance and open source. docs.google.com/forms/d/e/1FAI…
Want to easily scale MoE models like DeepSeek R1 with top performance and open-source? Join our vLLM Office Hours TODAY (Thursday) at 2PM EST. We'll show you how to scale MoE with @_llm_d_, a Kubernetes-native distributed LLM inference framework. RSVP: red.ht/office-hours
Distributed LLM inference is not hard. @_llm_d_ is a Kubernetes-native high-performance inference framework that makes it easy to get started. With llm-d v0.2, we’re laying down well-lit paths: reproducible, production-grade patterns for high-performance distributed inference...
⚡ Deploy distributed LLM inference on @digitalocean with llm-d KV-cache aware routing setup Disaggregated serving configuration Cost-effective DOKS deployment The future of LLM serving is distributed and open 🌊 🔗 se.isfusion.cloud/blog/llm-d-dok… Join us: llm-d.ai
🤝 Ready to build production LLM infra? llm-d's 7 SIGs welcome new contributors: ML Engineers → Benchmarking & Scheduling DevOps → Installation & Autoscaling Systems → Caching & Disaggregation SRE → Observability All meetings recorded and public. llm-d.ai/docs/community…
It’s hard to believe the @_llm_d_ project launched just 𝟮 𝗺𝗼𝗻𝘁𝗵𝘀 𝗮𝗴𝗼 — and it's already gaining serious traction across the industry!
🚀 Community win! Learn how to deploy llm-d on @digitalocean Kubernetes ✅ Step-by-step DOKS setup ✅ Core component deployment ✅ @vllm_project integration for high-perf serving Real practitioners sharing real deployment knowledge 🔥 👉 se.isfusion.cloud/blog/llm-d-dok… #llmd #vLLM
William Denniss @WilliamDenniss
1K Followers 219 Following Product Manager @GoogleCloud. Founder of GKE Autopilot. Author of Kubernetes for Developers https://t.co/xjqRqzqkyw
JillWallis @38jKKYzrJ0B3Ii7
30 Followers 791 Following
Chintan @is_future_
10 Followers 236 Following GPU Cluster, AI / ML on K8s / Openshift, Ansible - AAP, ArgoCD, ARO, AKS. DevOps, Setting up Infra at Scale
Matthew Bonanni @MatthewBonanni
119 Followers 145 Following PhD candidate @FXLabStanford | NASA Fellow | Researching combustion CFD
Nirjhar Jajodia @NirjharJajodia
3 Followers 37 Following
Andrew Smith @andrewjsmith00
17 Followers 240 Following
Thorsten Klein @IWillTry42
437 Followers 1K Following Software-Engineer & DevOps-Enablergiving with a love for OSS 🚀
Varoomer ⏩ @varoomer
1K Followers 3K Following ai/acc, core contributor @ https://t.co/vhrd7HiSrP building https://t.co/iPDpMtz7vh to arm the rebels in the race to AGI 🧠 ➡️ [email protected]
Dax @cloudmarooned
715 Followers 889 Following K8s hacker and open source enthusiast. Opinions created by me, usually poorly | ex-DevOps at @sourcegraph, ex-@Rancher_Labs
Kubernetes Podcast fr... @KubernetesPod
18K Followers 227 Following A news and interview show with insight from the Kubernetes community. Hosted by @kaslinfields and @boredabdel.
parameterfreak @parameterfreak
4 Followers 154 Following Thinking about model architecture and algorithms
Cedric Clyburn @cedricclyburn
690 Followers 619 Following Developer Advocate @RedHat • Organizer @KCDNewYork • Prev @MongoDB • containers, k8s, & everything in between
Victor Hugo @VictorHugo45995
0 Followers 7K Following
Lan Liang @lan31793328
81 Followers 165 Following CNCF ambassador @CNCFAmbassadors, a Kubernetes Contributor,Cloud Native Enthusiast| https://t.co/wMLZpdPHDd
Anton @25anton25
7 Followers 61 Following
samzong @samzong_
17 Followers 55 Following
Rahel Jhirad @RahelJhirad
2K Followers 7K Following Founder, Imaginator ai knowledge discovery 2D navigation TS ML DL recsys econ math incentives mech design finance networks bridges boundaries, Time, 3d type
Ran D @rlognormal
159 Followers 1K Following Software engineer with a fondness for structural molecular biology, LCMS proteomics, bio ml and llm workflows. DMs open for researcher collabs.
Alberto García L. @AlbertoGarLa
652 Followers 959 Following Software Engineer at @CoreOS @RedHat @IBM https://t.co/nXXeGWbBCp
Bikram Gupta @bikramkgupta
172 Followers 1K Following Builder. Everything infra, developer experience and security.
Hyunsik Choi @hyunsik_choi
445 Followers 613 Following Head of Platform SW at FuriosaAI (https://t.co/BFtrhQnUcZ), Member of Apache Software Foundation
mohinish.{eth|nft|nea... @mohinishbasha
267 Followers 2K Following Engineering @ KumoAI | ex @ DeltaStream, Confluent | Cloud & Realtime Data infrastructure | Kafka | Flink | K8 | Flux | Angel | Serverless Ventures
Randy @RandyNeu
416 Followers 4K Following Principal Cloud/AWS engineer bringing AI to the energy grid. Clean water, clean air, please. 'you can just do things should be on your headstone'
Mamading Ceesay @evangineer
3K Followers 6K Following Polymath. Doing Work That Matters (@H_O_L_O_), Enabling Thrivable Futures, Shaping Worlds, Smithing Words, Telling Stories, Playing Tabletop Games. He/Him/His.
Ragha Prasad @raghaprasad_r
103 Followers 443 Following Engineering @voltagepark. Opinions are my own.
Just Searching @jicinh
86 Followers 98 Following
Raymond Burkholder @oneunified
241 Followers 1K Following Software Developer, Open Source enjoyer, BGP AS10388 owner, Debian doer, data center network builder "Does each side of the story sum to the real story?"
Yuchen Fama @YuchenV2
8 Followers 29 Following 15 years of AI R&D and Product, amateur philosopher, art & nature enthusiast. Coding computer and model weights every day. Witness deeply, build bravely.
Ilari Stenroth @istenrot
662 Followers 4K Following
Ce Gao @gaocegege
7K Followers 786 Following Co-founder and CEO @TensorChord, building postgres-based vector extension https://t.co/7WGvl1sR56 | Father of 1 cat | Married
RANDRIAMANANTSOA @mihangy_har
1 Followers 74 Following
Mayank Bhaskar @cataluna84
1K Followers 4K Following Machine Learning Consultant 🧑🏽💻 | @twimlai & @Cohere_Labs Community Lead 👥 | @AILucknow ⌨ | #engineer 🛠 🧮 | #datavisualization 📊 | #sports ⚽ 🏓 🏋🏽 🎮
Dhananjay Kumar @dhnanjay
155 Followers 1K Following
Erkan Haspulat @erkanhaspulat
75 Followers 622 Following
skddmk @skddmk
16 Followers 1K Following
Safiullah_Niazi @Safiull93168968
116 Followers 1K Following Cyber Security Engineer || Automating Security Workflows || Incident Responder
Kubernetes @kubernetesio
314K Followers 87 Following #Kubernetes: open source production-grade container orchestration management. #CNCF #K8s
vLLM @vllm_project
17K Followers 20 Following A high-throughput and memory-efficient inference and serving engine for LLMs. Join https://t.co/lxJ0SfX5pJ to discuss together with the community!Trends for United States
You might like
