We train manipulation policies entirely in simulation. No human demonstrations, no real-robot training steps. A solver with full physics state generates the data; the policy only ever sees what a real robot’s cameras and encoders would see.

Current numbers. Single camera, behavioral cloning, 22k episodes: 94%. Two cameras, action chunking, 18k episodes: 99.25%.

Those go on the slide. The rest of this post is what it took, which was mostly a lot of runs that went nowhere.

Chunking on its own made things worse

Action chunking is everywhere now. Predict a short sequence of actions instead of one, cuts down compounding error. Obvious win. We added it and success went to 28.5%.

Took me a day and several runs to accept the ablation was telling the truth. Chunking on its own is not a weaker version of the published result. It’s a different thing that performs badly, and the gap between the two is entirely in how you handle the predictions once you have them.

That part I’m not going to detail yet. What I’ll say is that the piece everyone treats as an implementation detail turned out to be the piece doing the work, and finding that out cost us about $150 of A10 time.

Halving the resolution gave us exactly zero

Not “degraded.” Zero. Every trial, failed.

256×256 in works. 128×128 in and the policy does nothing useful. There’s a floor where the thing you need to see stops subtending enough pixels, and below it there’s no signal to learn from. What it looked like in practice was a failure to identify the target at all, followed by directionless driving. Not a near miss, not a wrong choice between two objects. No evidence it had located anything.

We found this by ablation. We could have found it in twenty minutes with a ruler and a pixel count. I’ve since done the measurement properly, which is its own post.

DAgger, twice

DAgger is the textbook answer to distributional shift. Roll out the student, ask the expert what it should have done in the states the student actually reached, retrain, repeat.

It made us worse. Both times.

I ran it a second time assuming I’d broken something in the first attempt. That’s usually the right assumption. Same result, same direction.

I still don’t have a clean explanation. Working theory is that DAgger assumes an expert who can label any state the student reaches, and ours can’t. The privileged solver is very good along its own solution manifold and undefined on states it would never have entered itself. So when the student wanders somewhere bad and we query for a correction, we’re not asking an expert what to do. We’re asking an optimizer to operate outside its basin. It answers confidently. We train on the answer.

That’s testable and I haven’t tested it. If it’s right, the failure should get worse the more aggregation rounds you run, which matches what we saw, but so would half a dozen other explanations. What we did was stop running DAgger and put the compute somewhere else, which felt like giving up at the time.

Validation loss lied to us for about a month

This is the expensive one.

Lower val loss, worse driving. Repeatedly. More than once the checkpoint that won on loss lost badly in closed-loop eval, and not by a little.

Makes sense once you say it out loud. BC optimizes per-step action prediction. What you care about is whether a few hundred sequential decisions compose into a rollout that works. Those aren’t the same objective. But knowing that in the abstract and actually not reaching for the loss curve when you’re tired and picking a checkpoint turn out to be different skills.

Closed-loop eval is slower, noisier, annoying to automate. Use it anyway.

The thing they have in common

I didn’t see it until I’d written them down next to each other.

In every case there was a check saying we were fine, and the check was built out of the same assumptions as the thing it was checking. Val loss is computed on data from the training distribution. Sim success is graded by the simulator that made the episodes. When the underlying assumption is off, the artifact and the check move together and the disagreement that would have told you never happens.

An oracle that shares an assumption with the thing it checks isn’t independent. It’s the same belief, wearing a different hat.

Not just a robotics thing

I read a JuliaHub eval last month. Frontier LLMs deriving physics models from a spec, graded against sealed ground truth. Different field entirely, same failure, and honestly they described it better than I have.

One model ran twenty-two separate validation checks, the most in the study, and shipped an answer thirty percent wrong. All twenty-two verified its own equations against themselves.

Another built an independent reference implementation, which is the right instinct, then gave the reference the same truncated time horizon as the model it was checking. Blinded its own check.

The one that scored highest deliberately corrupted a field component to confirm the check could detect something known-bad. That’s the move. Costs nothing. Almost nobody does it, me included until recently.

Which brings me to our own problem

Our teacher has privileged access: exact object poses, contact points, velocities, the lot. The student sees pixels and joint states. That asymmetry is the whole method.

But they’re both in MuJoCo.

Any place the simulator is wrong about the world, it’s wrong for both of them, identically. The student can match the teacher perfectly and score 99.25% on every sim metric we own and still be confidently wrong in exactly the shape MuJoCo is wrong. Our best result and our blind spot are the same number.

And by the argument above, no amount of additional sim-side evaluation catches it. They all inherit the assumption. I could run a thousand more eval episodes and learn nothing.

One unblinded oracle: the actual robot. It does the task or it doesn’t and nothing upstream gets a vote.

That’s what we’re running now, on a $50 RC car. Different platform to the arm the 99.25% came from, which is deliberate. If the method only transfers to the thing it was tuned on, it isn’t a method. I’ll post the number either way.

The Sim Gap. Fabian Friedland, Sim2Real Inc.

Every robotics CTO I speak to has the same confession.

They’re spending 3 to 5 times more on deployment than they budgeted.

Not on hardware. Not on the model. On making the robot actually work for customers.

We call it the sim-to-real gap. It’s the industry’s open secret. We all know it exists. For manipulation tasks, we’ve largely given up on training in simulation, preferring to generate data on actual hardware using teleoperation.

---

What Everyone Gets Wrong

The standard response to the sim-to-real problem is: build a better simulator (until then, stick to humans and hardware).

Better physics. More accurate friction models. Higher-fidelity rendering. More domain randomization.

This is the wrong answer.

Not because better simulators aren’t useful — they are. NVIDIA’s Isaac Sim is excellent infrastructure and we build on top of it. But the problem isn’t simulator accuracy. The problem is that we’ve been training robots to be precise when we should be training them to be robust.

A policy that learned to exploit a specific friction coefficient? Dead on arrival in the real world.

A policy that learned smooth, margin-tolerant behavior that doesn’t depend on simulation quirks? Usually fine — even in a mediocre simulator.

The field has spent 20 years building better simulators. We think that’s the wrong hill.

---

The Training Wheels Insight

Here’s the question that led us to found Sim2Real:

What if the way we generate training data is the problem — not just the simulator itself?

Current best practice for training robot policies is expensive and unscalable. You need a robot that already knows how to do the task — via teleoperation, motion capture, or human demonstration — to generate the data you need to train a robot to do the task. It’s the classic catch-22.

Our approach is different. For tasks we want to teach a robot, we first solve the problem with full access to the internal state of the physics engine — exact object positions, ground truth everything. It’s allowed to cheat. We call this “GOD mode”. The solution may be a simple Python script for easy tasks, or a trained neural network for more difficult behaviors (this is not the final neural net).

Think of it as training wheels: the GOD mode solution can see things a real robot never could, and it uses that information to solve the task perfectly under myriad initial conditions.

Then we run this solution millions of times with randomized environments. Different object positions, different lighting, different friction, different starting states. Each run is logged — but critically, we only log what a real robot can actually sense. Camera feed. Joint angles. Proprioception. No physics engine access.

The dataset generated that way — robot-legal observations paired with correct actions — becomes the training set for a neural network that can solve the task with only data available to the real robot.

So now we remove the training wheels.

We train the neural net on this “clean” dataset, without the physics engine access we used to create the data. This neural net has to learn to perform the task using only what it can sense in real life. The gap between what the “cheating” solution knows and what the final neural net can see is precisely where robustness is learned.

GOD mode solver: full physics access, perfect information. The deployed neural network never sees this data

Unlimited synthetic data. No human labeling. No teleoperation bottleneck.

---

Why This Works

The key insight is subtle but important.

When you give the robot privileged access to the physics engine, it finds the optimal solution for each randomized condition. When you strip that access from the neural net, it can’t rely on any specific physics assumptions — it has to generalize.

That forced generalization is deployment robustness.

The net doesn’t learn “pick up the cube when it’s at coordinates (x, y, z).” It learns “pick up the object that looks like this, from an approach direction like this, with a motion style that works across the range of conditions I was trained on.”

That’s a policy that survives contact with reality.

---

The Bigger Picture

The training wheels method is our near-term product. It solves the deployment problem today, using existing simulation infrastructure.

But we’re also working on something more ambitious.

Physics engines are, at best, an approximation. Even the best Newtonian engine we can build falls short of reality — because we are not God. The sim-to-real gap exists in part because no simulator perfectly captures the complexity of the real world.

What if the robot could define its own simulator?

Instead of building a better physics engine, let reality serve as the oracle. A correctly designed workflow could learn the dynamics that matter directly from real-world interaction, building a bespoke simulator tuned specifically to that robot in that environment. In the universe of all possible simulators, reality is the most accurate one.

That’s our moonshot. The training wheels method gets us to market. The learned simulator gets us to zero gap.

---

What We’re Building

This is the founding thesis of Sim2Real.

Three founders. Two of us — Dan Miller (CEO/CTO) and David Silver (COO) — previously co-founded On2 Technologies, which was acquired by Google. I’m Fabian Friedland, CSO, previously CEO of TychoBot where I spent two years deep in the sim-to-real problem before concluding the industry needed a fundamentally different approach.

We’re building the deployment layer for physical AI. Not a simulator — the methodology and tooling that makes simulation matter less.

---

The Sim Gap publishes weekly. The Zero Shot series goes deep on the technical foundations.

If you’re building robots, deploying models, or just tired of watching your policies fail on first contact with reality — subscribe.

---

Fabian Friedland is Chief Strategy Officer and co-founder of RealSIM (realsim.bot). He previously served as CEO of TychoBot and led Global Business Development at On2 Technologies (acquired by Google).

Subscribe now