Jia @basicv8vc

I like to fine-tune Deep Neural Nets on small datasets. Earth Joined January 2020

Tweets

28
Followers

41
Following

1K
Likes

3K

Jan Leike @janleike

2 years ago

Today we're releasing a tool we've been using internally to analyze transformer internals - the Transformer Debugger! It combines both automated interpretability and sparse autoencoders, and it allows rapid exploration of models without writing code.

17 188 1K 233K 838

Joshua Achiam @jachiam0

2 years ago

hot take: still true

Joshua Achiam @jachiam0

3 years ago

hot take: still true

1 1 19 0 2

9 5 102 24K 12

Rohan Paul @rohanpaul_ai

2 years ago

Interesting year ahead - Arthur Mensch, CEO of Mistral declared on French national radio that mistral will release an open source Gpt4 level model in 2024 radiofrance.fr/franceinter/po…

43 210 2K 315K 244

Soumith Chintala @soumithchintala

2 years ago

PyTorch's design origins, its connection to Lua, its intertwined deep connection to JAX, its symbiotic connection to Chainer The groundwork for PyTorch originally started in early 2016, online, among a band of Torch7's contributors. Torch7 (~2010-2017) These days, we also…

43 260 1K 465K 951

Aravind Srinivas @AravSrinivas

2 years ago

Tomas Mikolov, the OG and inventor of word2vec, gives this thoughts on the test of time award, and the current state of NLP, and chatGPT. 🍿

27 171 1K 335K 580

Download Image

Cas (Stephen Casper) @StephenLCasper

2 years ago

DPO might make LLM apps less secure if companies use user-sourced data to update them. I have tried and failed in the past to implant certain behaviors into ChatGPT with the user feedback feature. I think one of the reasons I failed is that it's hard to attack the policy…

2 6 12 3K 10

Nathan Lambert @natolambert

2 years ago

Another cool methods to add to the mess of PPO vs DPO vs KTO vs APA vs A LoL, I need more GPUs and more hands

Kawin Ethayarajh @ethayarajh

2 years ago

Another cool methods to add to the mess of PPO vs DPO vs KTO vs APA vs A LoL, I need more GPUs and more hands

19 128 683 169K 564

Download Image

2 7 31 5K 14

Kawin Ethayarajh @ethayarajh

2 years ago

📢The problem in model alignment no one talks about — the need for preference data, which costs $$$ and time! Enter Kahneman-Tversky Optimization (KTO), which matches or exceeds DPO without paired preferences. And with it, the largest-ever suite of feedback-aligned LLMs. 🧵

19 128 683 169K 564

Download Image

Albert Gu @_albertgu

2 years ago

Quadratic attention has been indispensable for information-dense modalities such as language... until now. Announcing Mamba: a new SSM arch. that has linear-time scaling, ultra long context, and most importantly--outperforms Transformers everywhere we've tried. With @tri_dao 1/

53 401 2K 803K 1K

Download Image

Jeremy Cohen @deepcohen

2 years ago

There is a discussion on ML reddit about a paper which shows loss curves for different models that seem to "line up." People were saying that it looks suspicious. Actually, in my experience, this is a real thing that happens when the data batch ordering is the same.

31 31 457 140K 138

Download Image

Yao Fu @Francis_YAO_

2 years ago

I have never understood reinforcement learning until

Zhengzhong Tu @_vztu

2 years ago

I have never understood reinforcement learning until

147 2K 13K 2.6M 2K

2 4 22 3K 6

Russ Salakhutdinov @rsalakhu

2 years ago

I hear a lot of folks in our AI community complain about openAI -- they don't publish, don't release models, maximum-for-profit, etc., so they are more like closedAI rather than openAI. This is true, but you have to give it to those guys -- they showed the true potential of LLMs…

42 66 876 209K 117

Andrej Karpathy @karpathy

2 years ago

Speculative execution for LLMs is an excellent inference-time optimization. It hinges on the following unintuitive observation: forwarding an LLM on a single input token takes about as much time as forwarding an LLM on K input tokens in a batch (for larger K than you might…

Georgi Gerganov @ggerganov

2 years ago

38 254 2K 1.2M 688

Download Video

106 596 4K 796K 2K

Susan Zhang @suchenzang

2 years ago

An interesting take from @Noahpinion on the possibility of generative AI becoming an equalizing force for access to higher-education/knowledge-work. Also worth going through Nicholas Carlini's survey/demo on what GPT-4 currently can/cannot do: nicholas.carlini.com/writing/llm-fo…

Noah Smith 🐇 @Noahpinion

2 years ago

12 11 121 142K 50

4 6 70 59K 47

Ben (no treats) @andersonbcdefg

2 years ago

As promised: Here's every article I could download as a .txt file from PubMed, up 'til August 18. Divided into: - commercial license: ~100GB - noncommercial license: ~60GB - other/unknown license: ~15GB - author-provided manuscripts: ~10GB Raw & unprocessed. Links below:

26 108 838 224K 1K

AI at Meta @AIatMeta

2 years ago

New on @huggingface — CoTracker simultaneously tracks the movement of multiple points in videos using a flexible design based on a transformer network — it models correlation of the points in time via specialized attention layers. 🤗 Try CoTracker ➡️ bit.ly/3swQFqt

38 420 2K 371K 640

Download Video

Peyman Milanfar @docmilanfar

2 years ago

A 1949 paper by Feller set the foundation for forward & reverse stochastic diffusion models, showing both can be described using the same functional form, and that the reverse process requires the score function. Except in @jaschasd's seminal 2015 work, it is hardly cited at all!…

4 68 442 80K 326

Download Image

(((ل()(ل() 'yoav))))👾 @yoavgo

2 years ago

I was puzzled for a while as to why we need RL for LM training, rather than just using supervised instruct tuning. I now have a convincing argument, which is also reflected in a recent talk by @johnschulman2 . I summarize it in this post: gist.github.com/yoavg/6bff0fec…

73 240 1K 353K 1K

Typst @typstapp

2 years ago

Typst has launched its beta! Register now on typst.app!

5 26 88 29K 32

Andrej Karpathy @karpathy

3 years ago

🔥 New (1h56m) video lecture: "Let's build GPT: from scratch, in code, spelled out." youtube.com/watch?v=kCc8Fm… We build and train a Transformer following the "Attention Is All You Need" paper in the language modeling setting and end up with the core of nanoGPT.