Markerless 3D Gesture-based Interaction for Handheld Augmented Reality Interfaces

Instead of physically touching the screen of mobile devices, natural gesture interaction can be an alternative input method for handheld Augmented Reality (AR). We present a novel interaction method that is based on identifying the positions and movements of the user's fingertips, and mapping these gestures onto corresponding manipulations of the virtual objects in AR environments.

With our prototype interface, users can hold the handheld device with one hand while controlling AR virtual objects with the other hand in front of the rgb-depth sensor. Specifically, users can use a pinch-like gesture with thumb and index fingers in six-degree-of-freedom (6DOF) in midair to manipulate virtual objects in 3D space on handheld AR devices. Users can be able to interact with virtual objects to modify their properties, like performing the translation, rotation or scaling to alter the location, pose or size of virtual objects in AR scenes.

Huidong Bai
The Human Interface Technology Laboratory New Zealand