The Deployment Problem Has a Simple Fix. Nobody Is Doing It.
Introducing The Sim Gap — and the idea behind RealSIM
Zero Shot #1
Every robotics CTO I speak to has the same confession.
They’re spending 3 to 5 times more on deployment than they budgeted.
Not on hardware. Not on the model. On making the robot actually work for customers.
We call it the sim-to-real gap. It’s the industry’s open secret. We all know it exists. For manipulation tasks, we’ve largely given up on training in simulation, preferring to generate data on actual hardware using teleoperation.
---
What Everyone Gets Wrong
The standard response to the sim-to-real problem is: build a better simulator (until then, stick to humans and hardware).
Better physics. More accurate friction models. Higher-fidelity rendering. More domain randomization.
This is the wrong answer.
Not because better simulators aren’t useful — they are. NVIDIA’s Isaac Sim is excellent infrastructure and we build on top of it. But the problem isn’t simulator accuracy. The problem is that we’ve been training robots to be precise when we should be training them to be robust.
A policy that learned to exploit a specific friction coefficient? Dead on arrival in the real world.
A policy that learned smooth, margin-tolerant behavior that doesn’t depend on simulation quirks? Usually fine — even in a mediocre simulator.
The field has spent 20 years building better simulators. We think that’s the wrong hill.
---
The Training Wheels Insight
Here’s the question that led us to found Sim2Real:
What if the way we generate training data is the problem — not just the simulator itself?
Current best practice for training robot policies is expensive and unscalable. You need a robot that already knows how to do the task — via teleoperation, motion capture, or human demonstration — to generate the data you need to train a robot to do the task. It’s the classic catch-22.
Our approach is different. For tasks we want to teach a robot, we first solve the problem with full access to the internal state of the physics engine — exact object positions, ground truth everything. It’s allowed to cheat. We call this “GOD mode”. The solution may be a simple Python script for easy tasks, or a trained neural network for more difficult behaviors (this is not the final neural net).
Think of it as training wheels: the GOD mode solution can see things a real robot never could, and it uses that information to solve the task perfectly under myriad initial conditions.
Then we run this solution millions of times with randomized environments. Different object positions, different lighting, different friction, different starting states. Each run is logged — but critically, we only log what a real robot can actually sense. Camera feed. Joint angles. Proprioception. No physics engine access.
The dataset generated that way — robot-legal observations paired with correct actions — becomes the training set for a neural network that can solve the task with only data available to the real robot.
So now we remove the training wheels.
We train the neural net on this “clean” dataset, without the physics engine access we used to create the data. This neural net has to learn to perform the task using only what it can sense in real life. The gap between what the “cheating” solution knows and what the final neural net can see is precisely where robustness is learned.
GOD mode solver: full physics access, perfect information. The deployed neural network never sees this data
Unlimited synthetic data. No human labeling. No teleoperation bottleneck.
---
Why This Works
The key insight is subtle but important.
When you give the robot privileged access to the physics engine, it finds the optimal solution for each randomized condition. When you strip that access from the neural net, it can’t rely on any specific physics assumptions — it has to generalize.
That forced generalization is deployment robustness.
The net doesn’t learn “pick up the cube when it’s at coordinates (x, y, z).” It learns “pick up the object that looks like this, from an approach direction like this, with a motion style that works across the range of conditions I was trained on.”
That’s a policy that survives contact with reality.
---
The Bigger Picture
The training wheels method is our near-term product. It solves the deployment problem today, using existing simulation infrastructure.
But we’re also working on something more ambitious.
Physics engines are, at best, an approximation. Even the best Newtonian engine we can build falls short of reality — because we are not God. The sim-to-real gap exists in part because no simulator perfectly captures the complexity of the real world.
What if the robot could define its own simulator?
Instead of building a better physics engine, let reality serve as the oracle. A correctly designed workflow could learn the dynamics that matter directly from real-world interaction, building a bespoke simulator tuned specifically to that robot in that environment. In the universe of all possible simulators, reality is the most accurate one.
That’s our moonshot. The training wheels method gets us to market. The learned simulator gets us to zero gap.
---
What We’re Building
This is the founding thesis of Sim2Real.
Three founders. Two of us — Dan Miller (CEO/CTO) and David Silver (COO) — previously co-founded On2 Technologies, which was acquired by Google. I’m Fabian Friedland, CSO, previously CEO of TychoBot where I spent two years deep in the sim-to-real problem before concluding the industry needed a fundamentally different approach.
We’re building the deployment layer for physical AI. Not a simulator — the methodology and tooling that makes simulation matter less.
---
The Sim Gap publishes weekly. The Zero Shot series goes deep on the technical foundations.
If you’re building robots, deploying models, or just tired of watching your policies fail on first contact with reality — subscribe.
---
Fabian Friedland is Chief Strategy Officer and co-founder of RealSIM (realsim.bot). He previously served as CEO of TychoBot and led Global Business Development at On2 Technologies (acquired by Google).