Joshua Gu @astrogu_
CS Phd student @MIT, @MIT_CSAIL, @MITEECSšØāš»| @LMCache Lab | Previous: BS @UChicago. Research on AI Systems Chicago, IL Joined December 2023-
Tweets30
-
Followers35
-
Following98
-
Likes24
LMCache supports gpt-oss (20B/120B) on Day 1! TTFT 1.20s ā 0.39s (-67.5%), finish time 15.70s ā 7.73s (-50.7%) compared to Vanilla vLLM. Release the true power of GPT-OSS with vllm+LMCache -- full deployment tutorial here: blog.lmcache.ai/2025-08-05-gptā¦ #LMCache #vLLM #OpenAI #LLMā¦
š Big news from LMCache Lab! š 3 papers accepted at SOSP ā25 & NSDI ā26, pushing the frontier of LLM-inference efficiency: 1ļøā£ Cross-agent KV-cache sharing (NSDI) š arxiv.org/abs/2411.02820 2ļøā£ Custom design for LLM prefillers (SOSP) š arxiv.org/abs/2505.07203 3ļøā£ā¦
such cool demo videos, wonder who made theseā¦ š¤
such cool demo videos, wonder who made theseā¦ š¤
š„ Check it out! š„
Excited to share our latest work š šš§šš¦ at #SOSP2025. This oneās special as itās my first full CS project from start to finishāfrom early brainstorming and iterating on ideas to running experiments and writing the paper. Learned a ton, and perseverance finally paid off! š
Excited to share our latest work š šš§šš¦ at #SOSP2025. This oneās special as itās my first full CS project from start to finishāfrom early brainstorming and iterating on ideas to running experiments and writing the paper. Learned a ton, and perseverance finally paid off! š
š¤
šØ LMCache now turbocharges multimodal models in vLLM! By caching image-token KV pairs, repeated images now get ~100% cache hit rate ā cutting latency from 18s to ~1s. Works out of the box. Check the blog: blog.lmcache.ai/2025-07-03-mulā¦ Try it š github.com/LMCache/LMCache #vLLM #MLLMā¦
š„³š„³š„³
š LMCache X @RedHat Official Collaboration LMCache is now a founding supporter of Red Hat's new llm-d project for scalable distributed LLM inference! š¤ Red Hat is also joining the LMCache project as an active contributor ā” Together we're building faster, more efficient openā¦
š LMCache turbocharges vLLM, KServe & Dynamo! Our new blog reveals how this SOTA KV cache layer slashes LLM inference costs & latency (TTFT/ITL). ā Massive scale (beyond single-node) ā Blazing speed with custom CUDA Kernels š Up to 2.3x prefill & 4.5x RAG throughput!ā¦
šš š¼š¼š»š°š®š°šøš² X šš šš®š°šµš²: KV Cache-centric Language Model Serving š We're thrilled to announce a strategic collaboration between LMCache and Mooncake to pioneer a KVCache-centric Large Language Model (LLM) serving system! This partnership is set to redefineā¦
š¤Æ 78.8% p95 Inter-Token Latency reduction with LMCache + vLLM v1 P/D support š In our previous blog, we introduced the integration of šš šš®š°šµš² with šššš šš and NVIDIA's š”šš«š library, enabling Prefill-Decode Disaggregation (PD) for LLM inference. Today, we'reā¦
š„ Tencent x @lmcache
š šš šš®š°šµš² Powers Up šššš š©š: P/D Disaggregation & NIXL Support! vLLM V1 revolutionized LLM serving, but lacked a dedicated KV cache interface for advanced optimizations... until NOW! ā” LMCache Lab is thrilled to announce two major updates enhancing vLLM V1'sā¦
š Exciting news from #EuroSys2025: Our work on CacheBlend won Best Paper! š CacheBlend delivers the first-ever speedup for RAG LLMs, achieving near-100% KV cache hit rates while maintaining output quality. Catch Jiayi Yao, Kuntai Du, and Shan Lu at EuroSys/ASPLOS this week &ā¦
FAST!!! š Thrilled to see all the hard work pay off!
FAST!!! š Thrilled to see all the hard work pay off!
š vLLM Production Stack is here!
š Deploying LLMs in Clusters #1 Check out this step-by-step tutorial to deploy the vLLM Production Stack on a cloud VM for superior performance and easy management. š Blog: blog.lmcache.ai/2025-02-13-cloā¦ š Code: github.com/vllm-project/pā¦ Comment for the topic next week! #k8s #vLLM
š„Meet the vLLM Official Production Stackš„ -ā”ļø 3x higher throughput & 3x faster response! -š§ Easy k8s deployment with helm chart! -š Observability dashboard! And itās open-source under vllm-project! Code: github.com/vllm-project/pā¦ Blog: blog.lmcache.ai/2025-01-21-staā¦ #LLM #vLLM #k8s
š LMCache speeds up multi-turn conversations by 7x vs. vLLM + prefix caching! Our secret? Efficient KV cache offloading to CPU, Disk, and even remote storage! šTry the benchmark now on your servers: github.com/LMCache/LMCachā¦ #LLM #Benchmark #vLLM #chat
AmandaJudith @1IU19A8s7YBP1F
1 Followers 267 Following
Ashutosh Maheshwari @asmah2107
24K Followers 1K Following Software Engineer @Youtube | Building LLM serving infra | AI | ex : @Google Search & @Microsoft Azure | 3x hackathon winner | Views my own
GalaxyAvaSmith @Ooxarsui797016
1 Followers 900 Following On the go and loving it Finding joy in the little things
Nadav Timor @NadavTimor
730 Followers 7K Following LLM inference, speculative decoding, open source. Built novel decoding algorithms ā default in Hugging Face Transformers (147k+ ā). Making LLMs faster + cheaper
Young Jack @YoungJack181308
0 Followers 12 Following
Dan Iskandarov @0xOnit3ngu
108 Followers 4K Following
Ajitesh Shukla @ajitesh_shukla7
1K Followers 6K Following Student,Love to solve hardest math problem. LLM's, Mathematical Research(Geometric Topology,Differential Geometry),Quantum Computing.Lord Krishna is God Of Math
cher @chenhao76566284
41 Followers 918 Following
Yichuan Wang @YichuanM
703 Followers 2K Following 1st year EECS PhD at UC Berkeley SkyLab @BerkeleySky, 2020 ACM class in SJTU, interested in MLSYS.
Bingyang Wu @wu_bingyang
7 Followers 959 Following
Rui Pan ę½ē @ruipeterpan
677 Followers 2K Following 3rd-yr PhD @PrincetonCS working on systems for ML/LLMs, interning @Google, previously @AmazonScience @maxplanckpress @WisconsinCS, fan of @fcbarcelona
victoriame @vmelnikova_en
872 Followers 619 Following the business of developer tools @evilmartians Dev Propulsion Labs podcast
LMCache Lab @lmcache
653 Followers 47 Following š§Ŗ Open-Source Team that maintains LMCache and Production Stack š¤ Democratizing AI by providing efficient LLM serving for ALL
Gudrun Lehner @LehnerGudr47620
50 Followers 4K Following
Qizheng Zhang @qizhengz_alex
264 Followers 510 Following CS PhD student @Stanford, Intern @MSFTResearch, @LMCache. Previous: BS @UChicago, @Argonne. Research on machine learning + systems.
Eejoewar @Eejoewar291231
0 Followers 53 Following
Shaoting Feng @shaoting_feng
13 Followers 41 Following Pre-Doc MPCS student @uchicago; Interested in system for ML
ę¦č¤ę¬åæ @JACKZJGJ
46 Followers 3K Following
Eric Yao @EricYao43125144
2 Followers 20 Following
fankero Š¤(k) ā A*t... @nofaro
90 Followers 1K Following
Nhan Ho @NhanHo033
0 Followers 246 Following
rick @rick11015787
2 Followers 136 Following
QI LIU @QILIU9203
2 Followers 66 Following
Lily-Kait @kaitomikik14643
75 Followers 7K Following
Elga @noharakiku35623
83 Followers 7K Following
Tetarnez @TetarnezaEfW
55 Followers 5K Following
CaraLena @7tIFo2zgi3dDR7J
67 Followers 7K Following
Junchen Jiang @JunchenJiang
402 Followers 319 Following CS Prof @ UChicago https://t.co/U01oOWGnip (Fast distributed LLM inference) https://t.co/hoetjwXKIt (Best KV cache layer)
Siddhant Ray @siddhantrayyy
131 Followers 198 Following Mainly Networks and Systems. Some Machine Learning. PhD CS candidate @ UChicago. MSc. EEIT @ ETH Zurich. BTech. ECE @ VIT Vellore.
lmarena.ai @lmarena_ai
92K Followers 207 Following LMArena: Open Platform for Community-driven AI Benchmarking. Graduated from UC Berkeley / @lmsysorg. Weāre hiring: https://t.co/1OkfLq2n0I
James Chen @jchencxh
849 Followers 507 Following mostly representation learning for vision and generalisation @SCSatCMU hot takes on everything else
Andrej Karpathy @karpathy
1.4M Followers 1K Following Building @EurekaLabsAI. Previously Director of AI @ Tesla, founding team @ OpenAI, CS231n/PhD @ Stanford. I like to train large deep neural nets.
Omar Khattab @lateinteraction
24K Followers 3K Following Asst professor @MIT EECS & CSAIL (@nlp_mit). Author of https://t.co/VgyLxl0oa1 and https://t.co/ZZaSzaRaZ7 (@DSPyOSS). Prev: CS PhD @StanfordNLP. Research @Databricks.
DSPy @DSPyOSS
10K Followers 48 Following An open-source declarative framework for building modular AI software. Programmingānot promptingāLLMs via higher-level abstractions & optimizers.
Shaoting Feng @shaoting_feng
13 Followers 41 Following Pre-Doc MPCS student @uchicago; Interested in system for ML
Yichuan Wang @YichuanM
703 Followers 2K Following 1st year EECS PhD at UC Berkeley SkyLab @BerkeleySky, 2020 ACM class in SJTU, interested in MLSYS.
Ajitesh Shukla @ajitesh_shukla7
1K Followers 6K Following Student,Love to solve hardest math problem. LLM's, Mathematical Research(Geometric Topology,Differential Geometry),Quantum Computing.Lord Krishna is God Of Math
Kuntai Du @this_will_echo
65 Followers 47 Following Chief Scientist | Committer of vLLM / LMCache / Production Stack
Hanchen Li @lihanc02
203 Followers 322 Following PhD student at @BerkeleySky working on AI Systems. alumni @uchicago, @lmcache
OpenAI Newsroom @OpenAINewsroom
109K Followers 3 Following The official newsroom for @OpenAI. Tweets are on the record. If you like this account, youāll love our blog: https://t.co/nEYf8Iq3C0
Eric Yao @EricYao43125144
2 Followers 20 Following
Tim Kraska @tim_kraska
3K Followers 274 Following Professor at MIT. Research director at AWS. Co-founder of https://t.co/0p4BiiYWYN and Instancio. Systems for ML / ML for Systems Research Blog: https://t.co/WUnjudXKHn
MIT EECS @MITEECS
28K Followers 308 Following MIT Department of Electrical Engineering and Computer Science ā we build the future.
Massachusetts Institu... @MIT
1.4M Followers 570 Following The Massachusetts Institute of Technology is a world leader in research and education. Related accounts: @MITevents @MITstudents @MIT_alumni
DeepSeek @deepseek_ai
973K Followers 0 Following Unravel the mystery of AGI with curiosity. Answer the essential question with long-termism.
Donald J. Trump @realDonaldTrump
108.8M Followers 53 Following 45th & 47th President of the United States of Americašŗšø
Nat Landau @realnatlandau
2K Followers 88 Following
Julia Landau @julia_landau
928 Followers 12 Following Daughter of Christopher Landau Groton 25 Duke 29
Christopher Landau @ChrisLandauUSA
215K Followers 562 Following Deputy Secretary of State, former US šŗšø Ambassador to Mexico š²š½. This is my personal account. Official account: @DeputySecState.
Tal Kachman @TalKachman
894 Followers 1K Following Assistant prof @DondersInst/@KachmanLab. Simulation and Coffee abuser. Ex:@TechnionLive--)@MIT--)@IBMResearch--)Rhizome--)@AQRCapital--)@AI_Radboud
Mimee // smart casual... @MimeeXu
625 Followers 335 Following what good can I do with my life if I only know math and computers. Doctorated (!) on ML x security/privacy.
arXiv math.PR Probabi... @mathPRb
1K Followers 0 Following Unofficial bot by @vela with https://t.co/qRyAuN2QxC. @mathMPb @mathNAb @mathNTb @mathOAb @mathOCb @mathQAb @mathRAb @mathRTb @mathSGb @mathSPb ...
Beidi Chen @BeidiChen
15K Followers 399 Following Asst. Prof @CarnegieMellon, @amazon Scholar, Prev: Visiting Researcher @Meta, Postdoc @Stanford, Ph.D. @RiceUniversity, Large-Scale ML, a fan of Dota2.
Zhuohan Li @zhuohan123
9K Followers 865 Following mts @ openai | cs phd @ š uc berkeley | building @vllm_project | machine learning system | the real agi is the friends we made along the way
Sam Madden @samrmadden
3K Followers 438 Following Professor, MIT EECS and Chief Scientist, Cambridge Mobile Telematics
Junchen Jiang @JunchenJiang
402 Followers 319 Following CS Prof @ UChicago https://t.co/U01oOWGnip (Fast distributed LLM inference) https://t.co/hoetjwXKIt (Best KV cache layer)
LMCache Lab @lmcache
653 Followers 47 Following š§Ŗ Open-Source Team that maintains LMCache and Production Stack š¤ Democratizing AI by providing efficient LLM serving for ALL
Yixin Wan @yixin_wan_
3K Followers 1K Following PhD student @UCLAComSci, Incoming Research Intern @TencentGlobal AI Lab | Trustworthy Generative Models | Previously @AmazonScience AGI, @MSFTResearch Asia
Ethan Mollick @emollick
288K Followers 576 Following Professor @Wharton studying AI, innovation & startups. Democratizing education using tech Book: https://t.co/CSmipbJ2jV Substack: https://t.co/UIBhxu4bgq
Markus Zimmermann @zimmskal
2K Followers 890 Following Benchmarking LLMs to check how well they write quality code. Support me using the profile link š
Prof. Anima Anandkuma... @AnimaAnandkumar
34K Followers 2K Following Bren Professor @caltech, Time100, Fmr Sr Director of #AI research @nvidia Fmr Principal Scientist @awscloud
Gappy (Giuseppe Paleo... @__paleologo
45K Followers 455 Following Curiosity, wonder, quantitative research. Books, read/written. Desire is that which is missing. Evil twin of @yogappygappy. new book: https://t.co/ygOgypsEQ5
Denny Zhou @denny_zhou
21K Followers 541 Following Founded & lead the Reasoning Team in Google Brain (now part of Google DeepMind). Build LLMs to reason. Opinions my own.
Joey Gonzalez @profjoeyg
4K Followers 409 Following Professor @UCBerkeley, co-director of @LMSysorg, and co-founder @RunLLM
Matei Zaharia @matei_zaharia
44K Followers 1K Following CTO at @Databricks and CS prof at @UCBerkeley. Working on data+AI, including @ApacheSpark, @DeltaLakeOSS, @MLflow, https://t.co/94gROE5Xa0. https://t.co/nmRYAKG0LZ
Song Han @songhan_mit
9K Followers 169 Following
Hao Zhang @haozhangml
6K Followers 474 Following Asst. Prof. @HDSIUCSD and @ucsd_cse running @haoailab. Cofounder and runs @lmsysorg. 20% with @Snowflake
Tri Dao @tri_dao
32K Followers 632 Following Asst. Prof @PrincetonCS, Chief Scientist @togethercompute. Machine learning & systems.
Francis Y. Yan @FrancisYan_
2K Followers 288 Following Assistant Professor at UIUC CS | Previously @MSFTResearch @Stanford | Creator of Puffer (400k users) | Illinois Networked Systems + AI Lab.
Rui Pan ę½ē @ruipeterpan
677 Followers 2K Following 3rd-yr PhD @PrincetonCS working on systems for ML/LLMs, interning @Google, previously @AmazonScience @maxplanckpress @WisconsinCS, fan of @fcbarcelona
Brett Adcock @adcock_brett
292K Followers 16 Following Founder @Figure_robot (AI Robots), @Cover_thz (Weapon Detection), @ArcherAviation (NYSE: ACHR), Vettery ($100M Exit)
OpenAI @OpenAI
4.3M Followers 3 Following OpenAIās mission is to ensure that artificial general intelligence benefits all of humanity. Weāre hiring: https://t.co/dJGr6Lg202
Trends for United States
You might like
