I've noticed there is some confusion about Dion since it mathematically looks so different from Muon and Spectral descent, so I wrote a small note expressing Dion in terms of the SVD and how it differs from PowerSGD š
Hyperplane projections are really powerful approach and seem to pop up everywhere in optimization. A quick overview which might be interesting to some š
When comparing optimization methods, we often change *multiple things at once*āgeometry, normalization, etc.āpossibly without realizing it.
Let's disentangle these changes. š
243K Followers 2K FollowingSign up for my new newsletter! (Link below) Also: Co-author of Abundance, host of Plain English, and contributing writer at The Atlantic.
88K Followers 2K FollowingWriting a data-driven newsletter about economics @ https://t.co/IanQ9oPoPi | Nuance? In this economy? | Full Employment Stan, Brazilian Coffee Tariff Victim
712K Followers 288 FollowingTogether with the AI community, we are pushing the boundaries of whatās possible through open science to create a more connected world.
50K Followers 403 Following@AnthropicAI. Prev. @Google Brain/DeepMind, founding team @OpenAI. Computer scientist; inventor of the VAE, Adam optimizer, and other methods. ML PhD.
20K Followers 452 Followingphysics of language models @ Meta (FAIR, not GenAI)
šļ¼Tsinghua Physics ā MIT CSAIL ā Princeton/IAS
š ļ¼IOI x 2 ā ACM-ICPC ā USACO ā Codejam ā math MCM
11K Followers 3K FollowingSenior director of Cisco Foundation AI, Former Chief Scientist at Robust Intelligence. ex Professor at Yale University, ex staff research scientist at Google.
950K Followers 764 FollowingProfessor at NYU. Chief AI Scientist at Meta.
Researcher in AI, Machine Learning, Robotics, etc.
ACM Turing Award Laureate.
1.2M Followers 279 FollowingWeāre a team of scientists, engineers, ethicists and more, committed to solving intelligence, to advance science and benefit humanity.
1K Followers 891 FollowingInterested in things that generalize. Currently RS @Meta, Prev: Science of Scaling co-TL @GoogleDeepmind. PhD Student at UC Berkeley. šŗšøšØš¦
7K Followers 652 FollowingResearch Scientist @AIatMeta
Previously Researcher @ Samsung AI
Outstanding Paper Award @icmlconf 2023
Action Editor @TmlrOrg
I tweet about ML papers and math
8K Followers 679 FollowingPhD student @MIT ⢠Research on Generative Models and Geometric Deep Learning for Biophysics ⢠BA @CambridgeUni ⢠Former @TwitterResearch, @DEShawGroup and @IBM
63K Followers 2K FollowingResearch Scientist at Google DeepMind (WaveNet, Imagen, Veo). I tweet about deep learning (research + software), music, generative models (personal account).
5K Followers 895 FollowingFaculty at @ELLISInst_Tue & @MPI_IS, leading the AI Safety and Alignment group.
PhD from @EPFL supported by Google & OpenPhil PhD fellowships.
1.4M Followers 1K FollowingBuilding @EurekaLabsAI. Previously Director of AI @ Tesla, founding team @ OpenAI, CS231n/PhD @ Stanford. I like to train large deep neural nets.
11K Followers 685 FollowingML theory nerd & AI non-enthusiast. thinking a lot about online learning these days!
BTW you should go find me on another website where i post more actively
58K Followers 619 FollowingDistinguished Professor (Emeritus), Oregon State Univ.; Former President, Assoc. for the Adv. of Artificial Intelligence; Robust AI & Comput. Sustainability
77K Followers 2K Followinga combination of a mediocre scientist, a mediocre manager, a mediocre advisor & a mediocre physicist at @nyuniversity (@CILVRatNYU) & @PrescientDesign