Mobile AR

    Free-Hand Interaction for Handheld Augmented Reality using an RGB-Depth Camera

    The goal of this research is to explore a novel interaction technique manipulating virtual objects in 3D space using bare hand on handheld Augmented Reality (AR) devices with RGB-Depth camera. Compared with conventional device-centric interaction methods like the keypad, stylus or touch-screen input, natural gesture-based interfaces offer a more intuitive experience for AR applications. Combining with the depth information, gesture interfaces are creating great opportunities to extend the handheld AR interaction to full 3D space. In this paper, we present a novel interaction method for handheld AR system implemented on a tablet with an external RGB-Depth camera attached. We retrieve the 3D hand skeleton from the color and depth frames, mapping the results to corresponding manipulations of virtual objects in the AR scene. Our proposed method allows users to control virtual objects using their bare hands in midair with canonical operations such as translation, rotation, and zooming.

    Huidong Bai
    Lei Gao
    Mark Billinghurst
    The Human Interface Technology Laboratory New Zealand
    University of Canterbury

    Jihad El-Sana
    Ben-Gurion University of the Negev