99.25% in Simulation Proves Nothing Yet
Four failures, and why every check we ran was blind to the same thing
Zero Shot #2
We train manipulation policies entirely in simulation. No human demonstrations, no real-robot training steps. A solver with full physics state generates the data; the policy only ever sees what a real robot’s cameras and encoders would see.
Current numbers. Single camera, behavioral cloning, 22k episodes: 94%. Two cameras, action chunking, 18k episodes: 99.25%.
Those go on the slide. The rest of this post is what it took, which was mostly a lot of runs that went nowhere.
Chunking on its own made things worse
Action chunking is everywhere now. Predict a short sequence of actions instead of one, cuts down compounding error. Obvious win. We added it and success went to 28.5%.
Took me a day and several runs to accept the ablation was telling the truth. Chunking on its own is not a weaker version of the published result. It’s a different thing that performs badly, and the gap between the two is entirely in how you handle the predictions once you have them.
That part I’m not going to detail yet. What I’ll say is that the piece everyone treats as an implementation detail turned out to be the piece doing the work, and finding that out cost us about $150 of A10 time.
Halving the resolution gave us exactly zero
Not “degraded.” Zero. Every trial, failed.
256×256 in works. 128×128 in and the policy does nothing useful. There’s a floor where the thing you need to see stops subtending enough pixels, and below it there’s no signal to learn from. What it looked like in practice was a failure to identify the target at all, followed by directionless driving. Not a near miss, not a wrong choice between two objects. No evidence it had located anything.
We found this by ablation. We could have found it in twenty minutes with a ruler and a pixel count. I’ve since done the measurement properly, which is its own post.
DAgger, twice
DAgger is the textbook answer to distributional shift. Roll out the student, ask the expert what it should have done in the states the student actually reached, retrain, repeat.
It made us worse. Both times.
I ran it a second time assuming I’d broken something in the first attempt. That’s usually the right assumption. Same result, same direction.
I still don’t have a clean explanation. Working theory is that DAgger assumes an expert who can label any state the student reaches, and ours can’t. The privileged solver is very good along its own solution manifold and undefined on states it would never have entered itself. So when the student wanders somewhere bad and we query for a correction, we’re not asking an expert what to do. We’re asking an optimizer to operate outside its basin. It answers confidently. We train on the answer.
That’s testable and I haven’t tested it. If it’s right, the failure should get worse the more aggregation rounds you run, which matches what we saw, but so would half a dozen other explanations. What we did was stop running DAgger and put the compute somewhere else, which felt like giving up at the time.
Validation loss lied to us for about a month
This is the expensive one.
Lower val loss, worse driving. Repeatedly. More than once the checkpoint that won on loss lost badly in closed-loop eval, and not by a little.
Makes sense once you say it out loud. BC optimizes per-step action prediction. What you care about is whether a few hundred sequential decisions compose into a rollout that works. Those aren’t the same objective. But knowing that in the abstract and actually not reaching for the loss curve when you’re tired and picking a checkpoint turn out to be different skills.
Closed-loop eval is slower, noisier, annoying to automate. Use it anyway.
The thing they have in common
I didn’t see it until I’d written them down next to each other.
In every case there was a check saying we were fine, and the check was built out of the same assumptions as the thing it was checking. Val loss is computed on data from the training distribution. Sim success is graded by the simulator that made the episodes. When the underlying assumption is off, the artifact and the check move together and the disagreement that would have told you never happens.
An oracle that shares an assumption with the thing it checks isn’t independent. It’s the same belief, wearing a different hat.
Not just a robotics thing
I read a JuliaHub eval last month. Frontier LLMs deriving physics models from a spec, graded against sealed ground truth. Different field entirely, same failure, and honestly they described it better than I have.
One model ran twenty-two separate validation checks, the most in the study, and shipped an answer thirty percent wrong. All twenty-two verified its own equations against themselves.
Another built an independent reference implementation, which is the right instinct, then gave the reference the same truncated time horizon as the model it was checking. Blinded its own check.
The one that scored highest deliberately corrupted a field component to confirm the check could detect something known-bad. That’s the move. Costs nothing. Almost nobody does it, me included until recently.
Which brings me to our own problem
Our teacher has privileged access: exact object poses, contact points, velocities, the lot. The student sees pixels and joint states. That asymmetry is the whole method.
But they’re both in MuJoCo.
Any place the simulator is wrong about the world, it’s wrong for both of them, identically. The student can match the teacher perfectly and score 99.25% on every sim metric we own and still be confidently wrong in exactly the shape MuJoCo is wrong. Our best result and our blind spot are the same number.
And by the argument above, no amount of additional sim-side evaluation catches it. They all inherit the assumption. I could run a thousand more eval episodes and learn nothing.
One unblinded oracle: the actual robot. It does the task or it doesn’t and nothing upstream gets a vote.
That’s what we’re running now, on a $50 RC car. Different platform to the arm the 99.25% came from, which is deliberate. If the method only transfers to the thing it was tuned on, it isn’t a method. I’ll post the number either way.
The Sim Gap. Fabian Friedland, Sim2Real Inc.