LIBERO is the simulation benchmark most robot model papers report. Researchers at UT Austin built it in 2023 to study robots that learn one task after another and carry what they learned over to the next (LIBERO). Its four standard suites have 10 tasks each, and each suite tests one thing: where objects stand, which object it is, what to do in the same kitchen, and two goals in a row. Every task comes with a scene, an instruction in plain language ("put the bowl on the plate") and a check that says when the goal is reached.
The usual way to run it is to record 50 human demonstrations per task and train a model on them. The best models then score around 97% (LIBERO-Plus, LeRobot). So LIBERO is saturated, the tasks are known up front and the scenes only shift a little. We didn't compete there. We used its scenes, tasks and checks in a whole new league: without demonstrations.
We ran all 40 tasks on Vesper with zero human programming, nobody writing or tuning code. The Vesper agent wrote each program on LIBERO's first starting position, the program then ran unchanged from 10 other starting positions and passed every time.
Spec in, program out
In Vesper we take in the task specification and develop the robot program from it. LIBERO contains exactly that: the scene, the prompt and the evaluation. A plant writes the same down for a new station: the cell, what the robot has to do and how acceptance is checked. So the benchmark suits us, we just approach it a bit differently. Where the models learn each task from demonstrations, the Vesper agent reads the task and generates the program.
With one camera, Vesper finds the objects, plans the motion, moves the arm and checks the result. No language model runs while the robot moves.
Without a human in the loop it scales
Zero demonstrations and zero programming take the human out of the loop completely. We were given virtual scenes, and those can be created on the fly or reconstructed from a real cell. The Vesper agent writes the program, runs it in simulation and fixes it until the check passes, and only then does it go to the robot. A new task costs compute time, which makes this massively scalable.
One program per task is what a plant wants
LIBERO is laid out to test one model that does every task. Our game is intrinsically different: the program has to be predictable and tailored to the task description. A task can still be broad. One program can branch on what the camera sees, for example which object comes in, and run the right sequence for it. In manufacturing you don't want unpredictable control. As long as reprogramming is dead cheap, a program per task is a strong asset for safety and reliability.
We'll be reporting on other benchmarks soon. First up is LIBERO-PRO, which swaps the objects, moves them further and rewrites the tasks. That is where generalization really gets tested.