Sim2Val: Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation.

Abstract

Learning-based robotic systems demand rigorous validation to assure reliable performance, but extensive real-world testing is often prohibitively expensive, and if conducted may still yield insufficient data for high-confidence guarantees. In this work we introduce Sim2Val, a general estimation framework that leverages paired data across test platforms, e.g., paired simulation and real-world observations, to achieve better estimates of real-world metrics via the method of control variates. By incorporating cheap and abundant auxiliary measurements (for example, simulator outputs) as control variates for costly real-world samples, our method provably reduces the variance of Monte Carlo estimates and thus requires significantly fewer real-world samples to attain a specified confidence bound on the mean performance. We provide theoretical analysis characterizing the variance and sample-efficiency improvement, and demonstrate empirically in autonomous driving and quadruped robotics settings that our approach achieves high-probability bounds with markedly improved sample efficiency. Our technique can lower the real-world testing burden for validating the performance of the stack, thereby enabling more efficient and cost-effective experimental evaluation of robotic systems.

Motivation

Traditional validation requires extensive real-world testing to achieve the confidence levels needed for safety assurances and certification
Simulation-only validation would be much cheaper, but simulators are not yet accurate enough for standalone validation, and would shift the problem to that of validating a simulator
Goal: Combine real-world and simulation testing to reduce real-world data requirements for validation

Method

Idea: Use simulation as a control variate!

With a control variate – a correlated signal whose expectation is known – we can reduce the variance of our estimator
Simulation measurements are correlated with real-world measurements
Because the true simulation mean is unknown, we estimate it from the sim-only data

Sim2Val estimates the true metrics mean by combining the paired real-world and simulation measurements.

Note that variance reduction is a function of the scale of sim-only data (k) and the correlation (ρ²) between real-world and simulation measurements.

If the original paired observations have low correlation:

We can learn a nonlinear metric correlator function (MCF) mapping scenario features + sim measurements → a refined surrogate metric
We can then use the new surrogate as a control variate:

Examples of Paired Scenarios

A real-world test (top) is reconstructed in simulation (bottom) to obtain a paired measurement.

Results

Sim2Val Autonomous Driving Performance

By leveraging inexpensive sim samples, we can achieve equivalent confidence level as using ~6x the number of expensive real-world samples! (Variance reduction of 82.9%)

Sim2Val Quadruped Velocity Tracking

Note that even the relatively modest reduction in variance from 2.048E−5 to 1.926E−5 would have required 38% more real-world tests without sim!

BibTeX


@inproceedings{luo2025_sim2val,
    title = {Sim2Val: Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation},
    author = {Rachel Luo and Heng Yang and Michael Watson and Apoorva Sharma and Sushant Veer and Edward Schmerling and Marco Pavone},
    booktitle = {Proceedings of the Conference on Robot Learning (CoRL)},
    year = {2025},
}