Dong Won (Don) Lee @_dongwonlee

🚀New from Meta FAIR: today we’re introducing Seamless Interaction, a research project dedicated to modeling interpersonal dynamics. The project features a family of audiovisual behavioral models, developed in collaboration with Meta’s Codec Avatars lab + Core AI lab, that…

Very happy to hear that GANs are getting the test of time award at NeurIPS 2024. The NeurIPS test of time awards are given to papers which have stood the test of the time for a decade. I took some time to reminisce how GANs came about and how AI has evolve in the last decade.

Can we use a single session level score for alignment align an LLM? Can we use multimodal signals to align a LLM? Yes, when you use them together! At #EMNLP2024, we present a new approach for aligning LLMs, called GELI 🍮, which can utilize extralinguistic, multimodal signals…

heading to #emnlp2024! would love to chat with those interested in joining our Multisensory Intelligence research group at MIT @medialab @MITEECS media.mit.edu/groups/multise… Our group studies the foundations of multisensory AI to create human-AI symbiosis across scales and sensory…

keyboard_arrow_left Previous keyboard_arrow_right Next

As an alternative to RLHF and adversarial training, we released short-circuiting. It makes models ~100x more robust. It works for LLMs, multimodal models, and agents. Unlike before, I now think robustly stopping models from generating harmful outputs may be highly tractable and…

keyboard_arrow_left Previous keyboard_arrow_right Next

New paper: The Platonic Representation Hypothesis In which we posit that _different_ foundation models are converging to the _same_ representation of reality. paper: arxiv.org/abs/2405.07987 website: phillipi.github.io/prh/ code: github.com/minyoungg/plat… 1/8

AI phone agents are here. Here are 8 startups enabling this you should know about + funding: 1. Bland AI - undisclosed - available as an API - $0.09/minute

New paper from @RekaAILabs 🔥 (yes an actual paper). This time we're releasing part of our internal evals which we call Vibe-Eval 😃 This comprises of a hard set which imo is pretty challenging for frontier models today. The fun part here is that we constructed it by trying to…

New paper on LLMs+culture! 🎊🎉 Thrilled to share our work on NormAd, a dataset evaluating whether LLMs can adapt to the diversity of cultural norms worldwide! (Spoiler: they can't!) ArXiv: arxiv.org/abs/2404.12464 w/ @akhila_yerukola @vishwayvs @_doctor_kat @MaartenSap [1/n]

I'm excited to share my recent publication in CHIL 2024, "Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Data". Our study reveals the potential of LLMs as personal health learners with wearable sensors. Arxiv: arxiv.org/pdf/2401.06866…

keyboard_arrow_left Previous keyboard_arrow_right Next

1/n A Ontology for Agentic AI Single Agent Architectures planning, self-correction, and suitability for straightforward tasks. Examples: * ReAct (Reason + Act): Iterative process of thought, action, and observation. * RAISE: ReAct with memory mechanism (short-term and…

AlphaLLM: An LLM that Learns and Improves Itself Large Language Models (LLMs) have revolutionized the field of Natural Language Processing, demonstrating remarkable capabilities in various tasks. However, they still struggle with complex reasoning and planning, often requiring…

Someone just dropped a dataset of 15 trillion tokens (as many as were used to train Llama 3)!!! Download this now before it gets taken down for “copyright reasons” Breakdown in thread 🧵 👇👇

Curious about socially-intelligent AI? Check out our paper on underlying technical challenges, open questions, and opportunities to advance social intelligence in AI agents: Work w/ @lpmorency, @pliang279 📰Paper: arxiv.org/abs/2404.11023 💻Repo: github.com/l-mathur/socia… 🧵1/9

I put together a quick colab notebook using Llama-3-8B-Instruct for chatting with a PDF. Unstructured API for partitioning and chunking a large PDF file, FAISS for vector storage, @LangChainAI for RAG, and quantized Llama-3-8B-Instruct (so that it fits on free Colab's GPU).…

We have a new preprint out - your language model is not a reward, it’s a Q function! 1. The likelihood of the preferred answer must go down - it’s a policy divergence 2. MCTS guided decoding on language is equivalent to likelihood search on DPO 3. DPO learns credit assignment

Can GPT-4V and Gemini-Pro perceive the world the way humans do? 🤔 Can they solve the vision tasks that humans can in the blink of an eye? 😉 tldr; NO, they are far worse than us 💁🏻‍♀️ Introducing BLINK👁 zeyofu.github.io/blink/, a novel benchmark that studies visual perception…

AK @_akhaliq

a year ago

Can GPT-4V and Gemini-Pro perceive the world the way humans do? 🤔 Can they solve the vision tasks that humans can in the blink of an eye? 😉 tldr; NO, they are far worse than us 💁🏻‍♀️ Introducing BLINK👁 zeyofu.github.io/blink/, a novel benchmark that studies visual perception… https://t.co/swelqZoue5

4 91 367 113K 165

Download Image

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments The first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating…

Learning social skills is out of reach for most people🙁 How can we make social skill training more accessible? We introduce 🌟APAM🌟 (AI Partner and AI Mentor) that leverages LLMs for social skill training via realistic practice and tailored feedback!

AK @_akhaliq

a year ago

Learning social skills is out of reach for most people🙁 How can we make social skill training more accessible? We introduce 🌟APAM🌟 (AI Partner and AI Mentor) that leverages LLMs for social skill training via realistic practice and tailored feedback!

2 51 234 121K 162

Download Image

We are super proud of Denison Guvenoz and Avery Kruppe, who presented their work, "Evaluating Robots' Social Intelligence", at the Purdue Fall Undergrad Research Expo! This marks the first public presentation by the HAI Group; keep up the great work team!