RoboGen

We present RoboGen, a generative robotic agent that automatically learns diverse robotic skills at scale via generative simulation. RoboGen leverages the latest advancements in foundation and generative models. Instead of directly using or adapting these models to produce policies or low-level actions, we advocate for a generative scheme, which uses these models to automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision. Our approach equips a robotic agent with a self-guided propose-generate-learn cycle: the agent first proposes interesting tasks and skills to develop, and then generates corresponding simulation environments by populating pertinent objects and assets with proper spatial configurations. Afterwards, the agent decomposes the proposed high-level task into sub-tasks, selects the optimal learning approach (reinforcement learning, motion planning, or trajectory optimization), generates required training supervision, and then learns policies to acquire the proposed skill. Our work attempts to extract the extensive and versatile knowledge embedded in large-scale models and transfer them to the field of robotics. Our fully generative pipeline can be queried repeatedly, producing an endless stream of skill demonstrations associated with diverse tasks and environments.

RoboGen is simulated and rendered with Genesis, a multi-material multi-solver generative simulation engine for general-purpose robot learning.

Change lamp light direction

Pull lever to start coffee brewing

Store the toy into the storage

Stand upright and walk forward using only hind legs

Bend noodle into a U-shape

Retrieve the gold bar from safe

Flatten the rice ball

Uncover trashcan lid

Close the drawer of the table

Heat up a bowl of soup in microwave

Kick the soccer ball to the left

Spin clockwise without right hind foot touching the ground

Retrieve the toy car from box

Slide in table drawer

Store the apple in refrigerator

Set the clock back by 5 minutes

Lie face down and cawl forward

Slide window halfway

Load dish into dishwasher

Stay balanced and move forward

Arrange three different cans in a row

Unload the milk from cart

Open the door of trash can

Take out a bottle of water from fridge

Press and rotate dispenser lid

Put a book in the drawer

Cut the dough in half

Rotate globe horizontally

Store a toy inside the box

Jump as high as possible

Put an toy into the storage

Direct and press dispenser

Kick the soccer ball to the left

Open washing machine door

Activate water faucet

Open fridges freezer door

Jump higher than 5 meters

Spin left without using right hind leg

Run backward fast using only front and left hind legs

Take a big jump backward

Pull out the top drawer of the cabinet

Press the button to turn on the printer

Push the ball forward

Open box lid halfway

Push the ball forward

Wrap the dumpling wrapper

Crawl backward at 1m/s

Start dishwasher by pressing the start button

Opening both refrigerator doors

Shape the dough into a baguette

Jump over the hurdle

Slide down display screen

Put a fruit into a fruit bowl

Jump and kick the basketball

Turning on coffee machine using the knob

Turn rightward continuously

Kick the soccer ball to the right

Extend suitcase handle

Lift up the dumpling

Stand upright on two hind legs

Rotate bucket handle

Do a backflip and kick the soccer ball to the right

Spin counter-clockwise

Put filling onto the wrapper

Turn on lamp by pressing the toggle button

Move clock ahead for daylight saving

BibTeX

@misc{wang2023robogen,
      title={RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation}, 
      author={Yufei Wang and Zhou Xian and Feng Chen and Tsun-Hsuan Wang and Yian Wang and Katerina Fragkiadaki and Zackory Erickson and David Held and Chuang Gan},
      year={2023},
      eprint={2311.01455},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation

Abstract

RoboGen Pipeline

Long-horizon task generation and learning

RoboGen generated tasks, scenes, training supervisions, and learned skills

Gallery

BibTeX