🤖 NVIDIA + APPLE = Smarter Robots

Creating robot training data with Apple's Vision Pro

Lex Sokolin
August 05, 2024

Hi Futurists,

Today we’re talking about Nvidia’s use of the Apple Vision Pro to record humans controlling robots for more realistic training data 👀

Remember, this week we are going to change up the domain from which you receive this email. Thank you for your continued support!

Best,
Lex

Accelerate AI Innovation with Gretel & Lambda Lab

Learn how Gretel and Lambda Lab unlock faster experimentation so teams can easily vet approaches, fail fast, and be much more agile in delivering a LLM solution that works. In this webinar, you will learn how to use Lambda Labs to fine-tune an SLM on several versions of the synthetic dataset from Gretel to reinforce how easy it is to experiment with task-specific LLMs.

Highlight of the Day

Transforming Robot Training with Apple's Vision Pro: Nvidia's Project Gr00t

S ource

Nvidia is pushing the boundaries of humanoid robot development with its latest initiative, Project Gr00t, aiming to bridge the "simulation gap" by utilizing Apple's Vision Pro headset for more realistic training data. This approach uses the headset to gather high-quality data by allowing humans to control robots from a first-person perspective, performing tasks such as making toast or retrieving a glass. As Jim Fan, Nvidia’s Senior Research Manager for Embodied AI, notes, “Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real-time.”

The magic happens when this human-generated data is fed into Nvidia’s RoboCasa simulation framework. The MimicGen system generates new actions and filters out unsuccessful attempts. This method, as Fan describes, trades "compute for expensive human data by GPU-accelerated simulation," effectively breaking the barrier of limited teleoperation data collection.

This technique could significantly narrow the sim-to-real gap — the challenge of translating robot training from simulation to the complexities of the real world. At the recent Siggraph conference, Nvidia CEO Jensen Huang highlighted the "three-computer problem" in robotics, underscoring the need for separate systems to create, simulate, and deploy AI. This comprehensive approach ensures robust development and optimization before real-world application, promising a new era in humanoid robotics.

Summary

Data Collection: Nvidia uses Apple's Vision Pro for gathering first-person perspective data.
Data Multiplication: RoboCasa framework exponentially expands collected data.
Action Generation: MimicGen system creates new actions and filters out failures.
Goal: Bridge the sim-to-real gap in robotic training.
Comprehensive Approach: Nvidia CEO highlights the "three-computer problem" for thorough AI development.

What we think

Nvidia's integration of Apple’s Vision Pro into Project Gr00t is a game-changer for a way to train humanoid robots. The exponential data multiplication via RoboCasa and MimicGen could lead to significant advancements in how robots learn and interact with the real world, potentially accelerating the deployment of AI in everyday tasks.

It is weird though to use tech like Vision Pro to put ourselves in the seat of robots, only to train and make robots in our image.

👉 Read More