DeforMe: Projection-based Visualization of Deformable Surfaces using Invisible Textures

The goal of our project is to realize interactive deformation visualization of a deformable object through enhancing the limitation of current projection-based mixed reality (MR) technologies, which support only rigid objects. In particular, we focus on the real-time estimation of "planar deformation", which is the tangential deformation of the surface such as a human skin, clay, and flubber, as realistically as possible. Once it is achieved, a user can interact with superimposed graphics directly through deforming the object, which is projected onto the surface to visualize the deformation. We also try to estimate subtle surface deformations so that we can visualize the self-dilation of a deformable object caused by the gravity force. Since deformable surfaces ubiquitously exist in our daily spaces, the proposed technique has a potential to be applied to various application fields, such as interactive clay toy, scientific visualization, user interface on user's body, product design, and so on. For example, we can provide an interactive clay toy application where a user can create a figure by clay on which dynamic graphics is projected to enhance the appearance of the clay and interactive. As an example of the scientific visualization, we can support hydromechanics engineers to assess the deformation of a material from different aspects by interactively visualizing various physical properties one by one with projected imagery. Furthermore, we can create a novel ubiquitous user interface in which a user can manipulate a computer by touching and deforming his/her skin. In summary, our contribution is to expand the possibility of projection-based MR so that we can interact with digital information on immediate surfaces in our daily spaces.

Parinya Punpongsanon
Osaka University

Daisuke Iwai
Osaka University

Kosuke Sato
Osaka University