From Real-World Space to Robot Deployment: Closing the Real2Sim-to-Sim2Real Loop
How can robots train and be evaluated against their destination before operating there? This article traces the complete R2S2R workflow, from reality capture and digital twins to simulation, deployment and field feedback.
Robots do not ultimately operate in idealized labs. They work in factories, warehouses and public spaces shaped by real scale, materials, lighting, obstacles and ongoing change. R2S2R begins by bringing the destination into the training environment, then returns evaluated policies to the field and uses operational data to refine both the scene and the task.
The R2S2R workflow in YUANQEE demos
These videos show YUANQEE work-in-progress demos of our spatial training workspace and ROS2 / NVIDIA Isaac Sim integration. They illustrate the technical workflow and do not imply support for every robot, sensor or operating environment.
Spatial Training Studio: scenes, robots and task authoring
Place reconstructed environments, robot models, sensors and task goals within a shared spatial frame for training-scene setup and task review.
ROS2 / Isaac Sim: simulation workflow validation
Validate scene integration, robot operation and task feedback through ROS2 and NVIDIA Isaac Sim as a foundation for adaptation to real deployment sites.
Why start with the destination
Generic synthetic scenes are useful for algorithm development, but they rarely capture the place where a robot will actually work. Corridor width, reflective floors, glass, thin structures, temporary clutter, changing light and sensor viewpoints can all affect localization, perception and navigation.
A training environment therefore needs more than visual realism. It must preserve task-relevant scale, geometry, materials, obstacle relationships and spatial reference. Digitizing the destination is part of deployment validation, not a visualization step at the end.
Real2Sim: bringing the real environment into simulation
Real2Sim starts with photos, video, 360-degree cameras, LiDAR or other multimodal data to reconstruct the environment where a robot will operate. The output is not only a browsable visual scene; depending on the task, it also needs scale-consistent, gravity-aligned and collision-ready geometry.
The visual layer supports camera perception, localization and scene understanding, while the collision layer supports path planning, contact reasoning and physics simulation. Deriving both from the same reconstruction and coordinate system reduces the misalignment introduced by manually rebuilt scenes and makes source data, versions and processing steps easier to review.
One spatial asset for training, localization and replay
Robot training involves more than importing a scene into a simulator. Teams must place the robot, sensors, goal poses, task objects and safety boundaries within the environment, while retaining the scene version, parameters and results for every run.
When training, localization, operational replay and scene updates share one spatial frame, failures can be reconstructed accurately: where the robot failed, what it perceived, what changed in the environment, and whether the next iteration should modify the scene, policy or task constraints.
Sim2Real: returning evaluated policies to the field
Sim2Real is not a one-time transfer of a policy to a robot. The policy should first be run, compared and screened repeatedly in a digital representation of the destination, followed by limited validation in the real environment. Trajectories, images, alerts and failure records from deployment should then feed back into the scene and evaluation workflow.
This loop turns deployment into continuous iteration: the real site supplies the constraints, simulation absorbs high-frequency experimentation, and field results shape the next training cycle.
What RGB, depth and spatial semantics each contribute
RGB preserves rich texture, color and lighting information for visual policies and semantic understanding. Depth and geometric data express distance, surfaces and traversable space more directly. Glass, reflective surfaces, low-texture areas, thin poles and cables can challenge different sensors in different ways.
A robust training environment rarely treats one modality as the sole source of truth. Instead, it aligns RGB, depth, point clouds, collision geometry and object semantics in one space, selects inputs according to the task, and records the provenance and limitations of every layer.
YUANQEE R2S2R: connecting spatial reconstruction with physical AI
YUANQEE defines R2S2R as Reality to Simulation to Reality: capture the destination, generate a trainable spatial scene, author and evaluate tasks in Spatial Training Studio and simulation tools, then return the results to robots and operational systems.
Our current focus is the end-to-end workflow across reality capture, 3DGS scene representation, geometry and coordinate processing, ROS2 / Isaac Sim integration, and task validation. 4DGS, open-vocabulary semantics, environment-change modeling and long-horizon task evaluation remain active R&D areas and will be introduced as validation progresses.
How to begin with a verifiable project
The first step is not to reconstruct an entire site. Choose one representative destination, one robot and one clearly defined task. First agree on the capture scope, robot and sensor configuration, success criteria, simulation interface and field constraints, then validate the workflow on a limited scale.
The result should answer three questions: Is the scene sufficient for the target task? Can simulation results explain field behavior? Can operational data be replayed in the same spatial frame? Expansion to more spaces, robot types and tasks should follow only when the evidence supports all three.
References and further reading
This article draws on the publicly released Real2Sim / Sim2Real methods and experiments from Niantic Spatial and Flexion. The underlying data, videos and results belong to their respective authors and organizations; YUANQEE cites them here solely for technical context.
Niantic Spatial, Flexion, and NVIDIA: Closing the Sim2Real Gap for Humanoids
Niantic Spatial and Flexion describe an end-to-end workflow spanning real-office reconstruction, training in Isaac Sim and deployment on a physical robot.