Sudhir Arora @sudarora

Joined April 2009

Tweets

16
Followers

20
Following

419
Likes

22

Simran Arora @simran_s_arora

a week ago

Intra-sm, inter-sm, and cross-gpu overlapping within a single kernel for llama 70b! Rather than compilers to fuse ops, we explore using an interpreter. While interpreters are often considered slow, the large granularity of ML operations and low overheads of this impl. allow large…

Benjamin F Spector @bfspector

a week ago

7 48 323 76K 227

Download Image

0 12 126 14K 57

Simran Arora @simran_s_arora

2 weeks ago

Very excited to share that I've finished my phd @Stanford and will be joining @Caltech’s cms department as an assistant professor. Looking forward to working with students and colleagues on ml systems! Grateful to my amazing advisor and labmates @HazyResearch for the best time…

122 57 2K 191K 287

Download Image

Jyo Pari @jyo_pari

a month ago

We are ending strong with GPU Programming 🚀! 2 talks today back to back! First @exists_forall for intro to CUDA and then @simran_s_arora for Thunder Kittens 🐈! Today at: 1:00pm EST / 11:00am PT - scale-ml.org/bootcamp

2 51 416 28K 276

Download Image

Simran Arora @simran_s_arora

4 months ago

There’s been tons of work on KV-cache compression and KV-cache free Transformer-alternatives (SSMs, linear attention) models for long-context, but we know there’s no free lunch with these methods. The quality-memory tradeoffs are annoying. *Is all lost?* Introducing CARTRIDGES:…

3 34 294 30K 218

Download Image

Simran Arora @simran_s_arora

4 months ago

~ megakernels ~ are next 🚀 excited to share a single kernel that runs a full llama 1b forwards pass!

Benjamin F Spector @bfspector

4 months ago

~ megakernels ~ are next 🚀 excited to share a single kernel that runs a full llama 1b forwards pass!

34 145 888 377K 528

Download Image

0 4 59 4K 8

Simran Arora @simran_s_arora

7 months ago

BASED ✌️ turns 1! One year since its launch at NeurIPS 2023 — and it's helped shape the new wave of efficient LMs. ⚡️ Fastest linear attention kernels 🧠 405B models trained on 16 GPUs 💥 Inspired Mamba-v2, RWKVs, MiniMax Checkout our retrospective below!

3 60 108 22K 30

Simran Arora @simran_s_arora

10 months ago

happy thanksgiving!! we're excited to share new thunderkittens blog posts this week for your enjoyment over the holiday break. today, we're starting off with fp8 support!!! 1500 tflops in <95 lines of code 🔥

3 23 108 13K 21

Download Image

Simran Arora @simran_s_arora

11 months ago

Wish writing AI kernels was like writing PyTorch??? Enter ThunderKittens 0.002: for simpler, faster, more adorable AI kernels! We use TK to provide 10-40% faster attention backwards, CuBLAS-speed GEMMs, 8x faster state space models, 14x faster linear attentions – averaging <200…

9 39 279 61K 170

Download Image

Simran Arora @simran_s_arora

12 months ago

Want Llama 405B, but wish it scaled linearly in sequence length??? Enter LoLCATS: an efficient method for "turning Transformers to linear attention models", all on an academic budget!! We use LoLCATS to linearize the *full Llama 3.1 model family* for the first time – 20+ points…

9 89 652 100K 524

Download Image

Simran Arora @simran_s_arora

a year ago

Enjoyed attending STOC 2024 recently to share some of the theoretical underpinnings of our work on Zoology, Based, and Just Read Twice! It’s such a wonderful community! Sharing slides and an explainer blogpost we wrote in conjunction with the talk for those interested:

1 7 24 3K 5

Download Image

Simran Arora @simran_s_arora

a year ago

Excited to share Just read twice: going beyond causal language modeling to close quality gaps between efficient recurrent models and attention-based models!! There’s so much recent progress on recurrent architectures, which are dramatically more memory efficient and…

7 59 300 93K 173

Download Image

Simran Arora @simran_s_arora

2 years ago

Excited to release Based, an architecture that combines two✌️ simple, familiar, attention-like primitives – short (size-64) sliding window attention and softmax-approximating linear attention – to enable high quality and efficient inference! 💨 🚀 joint w/ @EyubogluSabri,…

13 89 419 88K 258

Download Image

Simran Arora @simran_s_arora

2 years ago

KV-cache got you down? Sharing Based✌️, a simple architecture built from PyTorch 101 building blocks (convolutions, linear attention). It gives exciting quality vs the modern architectures & its hidden state size is fixed, enabling 4.5x >throughput vs attn hazyresearch.stanford.edu/blog/2023-12-1…

8 56 247 55K 112

Download Image

Simran Arora @simran_s_arora

2 years ago

LMs can be expensive for document processing. E.g., inference over the 55M Wiki pages costs >$100K (>$0.002/1k toks)💰 We propose a strategy that reduces inference cost by 110x and can even improve quality vs. running inference over each doc directly! 💻 github.com/HazyResearch/e…

7 131 764 128K 491

Simran Arora @simran_s_arora

3 years ago

Popular frameworks for personal ML e.g. federated learning display a tension between privacy and quality. We introduce a simple new framework, FOCUS, based on shipping foundation models to private silos, guaranteeing **perfect secrecy** When and where does this work, if at all?

1 7 16 0 2

Simran Arora @simran_s_arora

4 years ago

Reasoning over both public and *private* data is necessary for personalized ML systems. But how can we use personal context without exposing it to the world? 🔑🔒 We explore this question in new work on personalized and private retrieval systems!! dl.fbaipublicfiles.com/concurrentqa/r…