Ben Evans @justsomecsnerd

When I train an RL agent to play soccer

The "Supervised Policy Learning for Real Robots" tutorial at #RSS2024 is now public! supervised-robot-learning.github.io We're trying to democratize the "folk knowledge" of training real robot policies, and cover the fundamentals of imitation, recent algorithms, & practical considerations.

diff History was accepted to ICML and the camera-ready version is now available on arXiv! arxiv.org/abs/2312.07540 We argue that LM agents should be tuned and prompted using "text delta" summaries of recent interaction -- a text analogue of optical flow See you in Vienna! 🇦🇹

Ulyana Piterbarg @ulyanapiterbarg

2 years ago

diff History was accepted to ICML and the camera-ready version is now available on arXiv! arxiv.org/abs/2312.07540 We argue that LM agents should be tuned and prompted using "text delta" summaries of recent interaction -- a text analogue of optical flow See you in Vienna! 🇦🇹

3 13 114 34K 63

Download Gif

Building robot intelligence requires high-quality robot data. But far too many tools to collect data are closed and custom-built. Open Teach is fully open-sourced and is: - Calibration-free - Supports multiple arms, hands, & mobile manipulators - Costs just $500

LLMs swept the world by predicting discrete tokens. But what’s the right tool to model continuous, multi-modal, and high dim behaviors? Meet Vector Quantized Behavior Transformer (VQ-BeT), beating or matching diffusion based models in speed, quality, and diversity. 🧵

Deep SSMs are great in modeling audio / language, but they really excel in modeling continuous, raw sensor data. Introducing Hierarchical SSMs (HiSS), a new framework that improves upon Transformers, RNNs, S4 & Mamba for a range of sensors (touch, accelerometers & IMUs). 🧵👇

Open-ended decision-making is 𝙝𝙖𝙧𝙙, even for SOTA LLMs. Tuning on demos and providing models with history can help, but prompts quickly become verbose. Excited to share our new work: 𝙙𝙞𝙛𝙛 𝙝𝙞𝙨𝙩𝙤𝙧𝙮. Instead of scaling up models or data, tune on *text deltas* 🧵 1/n

A simple way to improve Language Agents: feed in sequences of text diffs with text generated in previous timesteps. ✅ SOTA in NetHack neural models. ✅ Requires 1800x lesser data than prior SOTA. ✅ Also works on simpler BabyAI environments. diffhistory.github.io

Wouldn’t it be nice if you could bring a robot home, give it a video of your room, and immediately start asking it to move objects around? Turns out, now you can! Introducing OK-Robot, a zero-shot language-specified pick & drop system that we built with exactly ZERO training! 🧵

Excited to release OK-Robot, an open-vocabulary mobile-manipulator for homes. Simply tell the robot what to pick and where to drop it in natural language, and it will do it. Like: Me: "OK Robot, move the Takis from the desk to the nightstand" Robot: ⬇️

Meet Dobb·E: a home robot system that needs just 5 minutes of human teaching to learn new tasks. Dobb·E has visited 10 homes, learned 100+ tasks, and we are just getting started! Dobb·E is fully open-sourced (including hardware, models, and software): dobb-e.com 🧵

Creating a "unified" policy for a robot is uniquely challenging for two reasons: (a) data is scarce; (b) a single robot can only act sequentially. In PolyTask, we learn a single policy to solve many tasks in a lifelong fashion given *just 1 demonstration* for each task. 🧵

📢#𝗥𝗼𝗯𝗼𝗛𝗶𝘃𝗲 - a unified robot learning framework ✅Designed for genralizn first robot-learning era ✅Diverse (500 envs, 8 domain) ✅Single flag for Sim<>Real ✅TeleOper Support ✅Multi-(Skill x Task) realworld dataset ✅pip install robohive tinyurl.com/robohive 🧵👇

Was great to be a part of this! Dexterous robots are becoming more proficient all the time.

Lerrel Pinto @LerrelPinto

2 years ago

Was great to be a part of this! Dexterous robots are becoming more proficient all the time.

11 68 296 139K 112

Download Video

When did you last see a cool robot in your home? Unless you own a robot vacuum, the answer is likely never. Lerrel Pinto, one of this year's 35 Innovators Under 35, wants to change that. trib.al/RFMBe8Q

> Services used by only IST teams too slow > Realize it's cause pods in us-west-1 > Ask teams to migrate them to ap-south-1 > 6 months discussions > Resolution reached, Migration approved, Yay! > Mandates a new VPN for login though, np. > mfw it is now twice as slow cause the…

🎉 A majority of NYU researchers have signed a Union Authorization Card to form our union 🎉 We’re excited to join thousands of researchers across the country who have organized for a voice over our working conditions, including most recently at Sinai and Columbia here in NYC!

keyboard_arrow_left Previous keyboard_arrow_right Next

I-JEPA: Efficient method for Self-Supervised Learning of image features. No need for data augmentation, just masking. Joint embedding predictive architecture, not generative. And it's open source, of course. Blog: ai.facebook.com/blog/yann-lecu… Paper: arxiv.org/abs/2301.08243 Code &…

AI at Meta @AIatMeta

2 years ago

I-JEPA: Efficient method for Self-Supervised Learning of image features. No need for data augmentation, just masking. Joint embedding predictive architecture, not generative. And it's open source, of course. Blog: ai.facebook.com/blog/yann-lecu… Paper: arxiv.org/abs/2301.08243 Code &…

37 335 2K 746K 524

Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. tactile-dexterity.github.io 🧵👇

While we are going gaga over large models and big data, there is still incredible value left to extract in small models and data, especially in robotics. All the skills shown below were each trained with <1 min of human data and <20 min of online RL fast-imitation.github.io 🧵👇