HandsIn3D: Augmenting the Shared 3D Visual Space with Unmediated Hand Gestures

A collaboration scenario involving a remote helper guiding in real time a local worker in performing a task on physical objects is common in a wide range of industries including health, mining and manufacturing. An established ICT approach to supporting this type of collaboration is to provide a shared visual space and some form of remote gesture. The shared space and remote gesture are generally presented in a 2D video form. Recent research in tele-presence has indicated that technologies that support co-presence and immersion not only improve the process of collaboration but also improve spatial awareness of the remote participant. We therefore propose a novel approach to developing a 3D system based on a 3D shared space and 3D hand gestures. A proof of concept system for remote guidance called HandsIn3D has been developed. This system uses a head tracked stereoscopic HMD that allows the helper to be immersed in the virtual 3D space of the worker’s workspace. The system captures in 3D the hands of the helper and fuses the hands into the shared workspace. A user study has been conducted showing that the proposed 3D immersive interface is helpful for improving perception of spatial relation of helpers and sense of co-presence and that the system is generally useful and usable, particularly more so for complex tasks.

The unique feature of HandsIn3D is the integration of the projection of the helper’s hands into the 3D workspace of the worker. These hands and gesture projections provide rich, non-verbal communication which creates a sense of “working together of a helper and a worker side by side”. In reality, the helper and the worker can be separated by long distances.

Weidong Huang
Commonwealth Scientific and Industrial Research Organisation

Leila Alem
Commonwealth Scientific and Industrial Research Organisation

Franco Tecchia
PERCRO Perceptual Robotics Laboratory