Free-Hand Interaction for Handheld Augmented Reality using an RGB-Depth Camera

The goal of this research is to explore a novel interaction technique manipulating virtual objects in 3D space using bare hand on handheld Augmented Reality (AR) devices with RGB-Depth camera. Compared with conventional device-centric interaction methods like the keypad, stylus or touch-screen input, natural gesture-based interfaces offer a more intuitive experience for AR applications. Combining with the depth information, gesture interfaces are creating great opportunities to extend the handheld AR interaction to full 3D space. In this paper, we present a novel interaction method for handheld AR system implemented on a tablet with an external RGB-Depth camera attached. We retrieve the 3D hand skeleton from the color and depth frames, mapping the results to corresponding manipulations of virtual objects in the AR scene. Our proposed method allows users to control virtual objects using their bare hands in midair with canonical operations such as translation, rotation, and zooming.

Huidong Bai
Lei Gao
Mark Billinghurst
The Human Interface Technology Laboratory New Zealand
University of Canterbury

Jihad El-Sana
Ben-Gurion University of the Negev