# Robotics needs context engineering

Ruben Verhack, 2026-11-17. https://vesper.dev/blog/robotics-needs-context-engineering/

Playground just published [a nice study](https://www.withplayground.ai/blog/how-should-we-use-llms-for-robot-control) on how to use LLMs for robot control. A frontier model got ten LIBERO tasks. Writing a program from how the scene is built, without ever running it, it solved 88% of 500 unseen scenes. But when it only got the camera image and had to drive the robot live, the same model solved just half, at more than 50 times the cost per success.

Same model, same tasks. The difference is what it got to read: where the objects get placed, how high the table is, where the camera looks. That's information a plant mostly already has. Drawings, CAD, the layout of the cell, spec sheets, the ERP.

Better information gives better results, nothing shocking. Yet most physical AI approaches put their effort in measuring after the fact: teleoperation, demonstrations, hours of recorded data. A demonstration records what the operator did. It doesn't record what they knew when they did it: which variant it is, what it weighs, what the drawing says. The model has to reverse-engineer all of that from the movements.

Far less effort goes into cleaning up and prepping the context a robot program starts from. With LLMs we learned to call that context engineering, and robotics needs a lot more of it. It's another dimension to improve a robot program on, next to the model and the trial and error around it. For manufacturing, most of that context is already written down.

## We engineer what our agent reads

In Vesper, a coding agent writes the robot program from the task and what it knows about the cell, and the robot runs the result. That's [generative robot programming](https://vesper.dev/blog/agents-write-the-program/). When we [ran LIBERO on Vesper](https://vesper.dev/blog/robot-demonstrations-dont-scale/), our agent got a description of each scene and the task, no demonstrations.

That context is one more lever to pull. The model, the building blocks, the simulation, what the agent reads: we can improve each of them, application after application. Demonstration-based methods mostly have one lever, more data. They end up in a whack-a-mole situation where every failure means more demonstrations, and cleaning up and checking that data is tedious and hard.

## Better building blocks, less reliance on the model

Context doesn't cover everything. Whatever isn't known, the robot has to estimate from the camera while it runs: where the part is, how to approach it, how to grasp it. Recovery strategies and quality checks during the run belong there too. That's the job of the building blocks, and why they have to generalize.

The study also shows how much the model matters today. A second model, same rules, solved about a third. We want building blocks good enough that you rely less on the coding agent's abilities. Still, new frontier models will always push newer applications, and they leverage the same building blocks better too. So there will most likely always be a bit of performance loss with a weaker one.
