Large Time-Delay Teleoperation for Enabling Robotic Assembly on the Moon through Extended Reality and Machine Learning

Overview

Future lunar bases will rely on robots to assemble and maintain infrastructure, with human operators on Earth staying in the loop for key decisions. The Earth–Moon ~3s round-trip delay makes direct teleoperation unstable and mentally demanding, especially when the operator receives force feedback. At the same time, full autonomy cannot yet handle every situation in a partially known lunar environment.

LunarAssembly addresses this by combining immersive Extended Reality (XR), delay-robust haptic control, and robot skill learning in a shared autonomy framework. The robot learns assembly skills from human demonstrations and executes them on the remote site, while the operator can:

• preview and adjust a skill in XR before it is executed,
• take over and correct the robot at any moment during autonomous execution,
• fall back to direct force-feedback teleoperation that remains stable despite the delay, and
• perceive the remote scene as a 3D reconstruction with detected hazards, rather than only through 2D camera feeds.

Overview of the LunarAssembly framework, integrating teleoperation, shared autonomy, skill learning and XR-based teleoperation.

Research Objectives

Immersive XR Telepresence

The remote scene is reconstructed in 3D from the robot's LiDAR data and rendered in XR, giving the operator a navigable view of the work area. Rocks and craters are segmented using a deep learning model and displayed as 3D hazards, helping the operator cope with the low sun angles and long shadows expected at the lunar south pole.

Delay-Robust Haptic Teleoperation

LunarAssembly builds on the Time Domain Passivity Approach to keep force-feedback teleoperation stable under multi-second delays. The project extends it so that the required damping is spread over time and separated from the operator's own commands, so stability does not come at the cost of operator comfort.

Skill Learning and Shared Autonomy

Assembly skills such as picking, placing, and pressing parts into place are taught by demonstration in a delay-free lab setting and encoded as task-parameterized models that generalize to new object positions. On the remote site, these skills run autonomously, and the operator's corrections can be used to refine them incrementally.

Integration and Validation

The components are being integrated and tested in SpaceR's LunaLab, a lunar analogue facility with a basalt test bed and controllable lighting. A mobile manipulator has already been teleoperated between TU Wien in Vienna and the LunaLab in Luxembourg (~760 km apart) over the public internet. The next steps are to develop the XR framework with 3D reconstruction, integrate it with the teleoperation system, and begin testing under emulated Earth–Moon delays.

Consortium

University of Luxembourg — SnT / SpaceR
Project coordination, XR framework with 3D reconstruction and AI-based hazard detection, system integration, and experimental validation in the LunaLab.

TU Wien — Robotic Systems Lab
Delay-robust haptic control and passivity-based force-feedback teleoperation.

TU Wien — Autonomous Systems Lab
Robot skill learning from demonstration and shared autonomy.

Publications

• H. T. Dinc, J. Heidersberger, C. Kyriazis, T. Wagner, A. Frantz, D. Pisanti, F. Kong, M. A. Olivares-Mendez, D. Lee, and C. Ott. "Earth to Moon Bilateral Teleoperation for Lunar Assembly: A Unified Framework for Shared Autonomy, Skill Learning, and Telepresence." Accepted at the 2026 IEEE Conference on Telepresence.