AR in VR: Omnistereoscopic Telepresence with Holograms for Remote Maintenance and Collaboration

Abstract

With the development of mixed reality technologies, remote maintenance and collaboration applications receive increasing interest in the industry. The combination of 360-degree telepresence and augmented reality (AR) cues have been shown to be an effective way of remote collaboration. However, most existing methods do not provide depth perception, which can improve the remote inspection. Also, many methods that use holograms for live collaboration are limited with only pointing functionality. In this paper, we present a 360-degree telepresence system interacting with an AR backend to create an immersive mixed reality interface for remote collaboration. The proposed camera system does not need to be carried by any on-site person, but can be controlled by the remote operator intuitively. We deploy network delay compensation methods and propose novel projection strategies for a correct and efficient rendering. A comparative analysis shows advantages of the proposed system over previous work, and that it is a promising approach for improving remote collaboration and maintenance. Our experiments indicate that our system has a glass-to-glass delay of roughly 106 ms, and a data rate to the client varying between 10-15 Mbps, which can be further optimized for specific applications. Exploratory tests with VR-experienced users showed that the motion-to-photon latency of the system is in acceptable ranges. The system can be deployed in various industrial applications for a live or asynchronous collaboration between an on-site user (with AR) and an off-site user (with AR in VR).