Presence Through Fullbodied Interactions Overview of research activities at the Immersive Interaction Group

The Immersive Interaction Group aims at ensuring active presence through the transparent performance of complex full-body actions in potentially complex environments. One goal is to leverage on natural human skills regarding displacements, manipulation, coordination and adaptive control for the evaluation of virtual prototypes. An alternate goal is to improve the training of full body skills either for dangerous tasks or for rehabilitation. Finally a last implicit goal is to contribute to the proposition of more believable real-time autonomous virtual humans for the evaluation of complex interpersonal activities. To achieve these objectives we have to dedicate the computing resources to the immersive dimensions for which human psychophysical sensitivity is high. For this reason we will continue to carry on experimental studies on human attention and performance during interactions through a full body avatar.


I. LAB HISTORY AND MISSION
In February 2011, the Immersive Interaction Group (IIG), led by Dr Ronan Boulic, stemmed from the Virtual Reality Laboratory (VRLAB) founded by Prof. Daniel Thalmann in 1989.The VRLAB has been mainly involved in the modeling, animation and real-time interaction with inhabited 3D virtual worlds.It has contributed to the introduction of the virtual human technology in real-world applications such as the VR psychotherapy for social phobia.
Research projects conducted by Dr Ronan Boulic within the VRLAB have focused on a wide range of topics including the realistic modeling of locomotion allowing personification and continuous real-time speed variations [1], motion capture and full-body interactions [2], and the interactive postural control of virtual humans under conflicting constraints [3].This latter topic led to the proposition of off-line motion editing tools [4], the integration of data-driven motion model for guiding the Inverse Kinematics convergence [5] and real-time full-body interaction [6] possibly in cluttered virtual environments [7].
Last February 2008 the American National Academy of Engineering presented fourteen topics identified as the grand challenges of the 21st century that should be solved to improve the quality of life in the coming years.Among them, is the challenge "Enhancing Virtual Reality", which highlights the high potential of investigating what makes an immersive interaction a truly effective means of uncovering new knowledge [8].The mission of the new research group is clearly in line with this objective.Our goal is to make possible intuitive and transparent full-body interactions for potentially complex information spaces, be it a virtual prototype for high tech industries, a highly dynamic simulated train station bursting with commuters, or even an abstract space.The user must be able to react as if the virtual environment were real, even if it is not realistic.This is the general goal of Presence but our special focus is to take advantage of the user natural abilities for full-body movement so that the proposed VR techniques and tools recedes from awareness and become an extension of the user's body [9].

II. LAB OBJECTIVES
Our general goal brings in the well-known issue of realtime interactions: where should the computing resources be allocated to enforce and preserve the user involvement?To answer this question we plan to carry on complementary studies to better identify the psychophysical sensitivity of the interacting user to alterations in terms of spatial precision, choice of viewpoint, scale and co-location, especially when the interactions are mediated through full-body avatars (Fig. 1 a, b).
For example, we have recently addressed the topic of comparing various scaling strategies when embodying heightdiffering avatars [10].Such research is necessary when one want to evaluate the use of a virtual prototype for a large range of the population; in such context the VR setup must ensure that the engineer who is immersed in the virtual prototype environment feels as if he were a potentially completely different person (e.g. a child with a much smaller body height as illustrated on Fig. 1c and d).A second key objective is to pursue our past efforts in motion modeling (locomotion [11], reach [5]) through the setup of dedicated experimental campaigns so that users can interact with believable autonomous virtual humans.We will also continue to investigate complementary topics in emotion modeling [12].

III. RESEARCH LINES
Our central research focus is to provide computationally efficient numerical algorithms for facilitating real-time 3D interactions with any type of reality (virtual, mixed, augmented, information space, etc).These algorithms belong to the general family of mathematic methods for optimization, interpolation, approximation and numerical linear algebra.They are exploited within a general modeling/simulation methodology to solve application-oriented problems.Our research roadmap is structured along the following research lines.

A. Ensuring precise real-time full-body co-location
The user action is central to the VR experience we target.Indeed we want to leverage on the intrinsic user's motor skills to evaluate complex tasks within virtual prototypes.So our first research line is to continue to enhance real-time and precise full-body motion capture while being the least possible invasive.A recent comparative study shows that our approach is highly precise but at a high computing cost [13].Three directions for improvements will be explored: 1/ by taking inspiration from [6], further increase the precision of highly dynamic movements by combining analytic and Jacobianbased approaches within the Inverse Kinematics solver, 2/ enlarge the management of inequality constraints to address complex environments with concave obstacles (Fig. 2 illustrate the current state of our research with convex obstacles), 3/ minimize invasiveness without sacrificing precision by exploiting Kinect sensor input data instead of active optical markers.
Figure 2. The posture of the virtual human avatar is driven by motion capturing the user [8]; the posture reconstruction is combined with collision avoidance owing to damping constraints (red lines) [7].

B. Understanding avatar-mediated perception-action
Our second line of research is intimately linked to the first one in the sense that we want to uncover how sensitive the user is to a possible mismatch between the planned intentions and the perceived interaction.The numerous approximations in the motion capture process are causing such mismatches and it is important to identify and address in priority those that cause breaks in presence.An alternate source of mismatch might be introduced voluntarily to handle the automatic collision avoidance (Fig. 2) so that the user does not have to care about it by making tedious postural adjustments.In this second case, we hypothesize that the user could be rather tolerant to such mismatches provided they do not occur in the body region in charge of a task.A complementary direction of investigation is to characterize whether or how the nature of the embodiment influences the user behavior in the virtual environment.Hints about significant behavioral differences were observed in [10] when the subjects were embodied with a full avatar or through only a rigid solid materializing the 3D location of the hands.In this line of research we will set up psychophysical experiments dedicated to the evaluation of precise perception-action coupling while controlling an avatar in a virtual environment.

C. Modeling of believable virtual humans
Numerous applications request the interaction with other humans being.Satisfying user expectations through believable interactions with virtual humans help to sustain the user motivation to engage in and pursue a proposed task.Hence it is necessary to rely on, and advance the understanding of fundamental human motion and emotion patterns involving the full-body such as locomotion, steering, reach, gaze, body language, etc… Their coherent integration into autonomous agents is critical for the success of a wide range of applications (serious games, urban planning, collaborative training, rehabilitation, etc).

D. Understanding avatar-mediated interactions with agents
As a complement to the last two lines of research, we feel it is crucial to characterize the influence of the user embodiment into an avatar when interacting with autonomous virtual humans.We hypothesize that the more the user is embodied through a fully controlled avatar body, the higher the physiological response and the sense of presence.The additional difficulty in our line of research is that the user is not a mere passive observer but is actively involved through movements, either task-oriented or resulting from the expression of body language.

IV. ONGOING PROJECTS
We provide first a brief overview of the available infrastructure followed by a short description of on-going projects in motion modeling, autonomous pedestrian agent (AerialCrowds), avatar-mediated communication with an emotion model (CyberEmotions), and space perception (VSPACE).

A. Infrastructure
Virtual Reality hardware facilities include a four screen immersive CAVE with stereo projection system (Fig. 3), a haptic workstation composed of twoCyberGloves and the CyberForce exoskeleton conveying force feedback to both arms (Fig. 4), an eye tracking system, three head mounted systems, and a physiological measurement system.The motion capture hardware consists of a Phasespace active optical motion capture system with 14 cameras and a 60 m2 motion acquisition space (Fig. 5).The motion capture system is mobile and can be used for off-line motion recordings (Fig. 5) or online head-tracking (Fig. 3) or full-body interaction (Fig. 1a).

B. Motion modeling: the stepping pattern
Local movements including only a few steps are frequent for a large range of activities and social behaviors.However traditional locomotion model are not suited for producing local displacements with potentially large reorientations.So we conducted a motion capture campaign to capture a dense set of stepping sequences.Our first goal was to compare some key features (number of steps and chosen support foot for each target) with the predictions made with an Inverse Kinematic model [15].Subject were instructed to start at a projected location as in Fig. 6 (left) in the capture area (Fig. 5) and to move to a constant end location in space (Fig. 6 middle).An example of feet and center of mass trajectories can be seen on Fig. 6 right.A partial analysis of the measurements highlights a great diversity among the five subjects in terms of number of steps for reaching the 150 targets.Predictions were consistent with data when the target was in front of the subject.On the other hand they were likely to make more steps than predicted when the target was behind.Further analysis of the data is necessary to characterize the criterion supporting the behavioral change.

C. AerialCrowds: reusing real pedestrian behaviors
In this project we want to immerse the user in a simulated crowd to evaluate a given environment under various contexts such as variable crowd densities, and adding/removing elements from the environment.We believe that the ecological validity of the crowd motion will be improved by exploiting real pedestrian trajectories captured in an outdoor area.Likewise, the pedestrian trajectories have to be obtained through a non-invasive means; for this reason we exploited multiple overlapping video cameras in collaboration with the CVLab from EPFL.
The project is in the stage of trajectory reuse for synthesizing new crowds in virtual environments that are completely different from the original location [16].Figure 7 illustrates the original capture location with reconstructed trajectories (left) and their reuse in a new environment (right).Future work will focus on embedding the user in the synthesized crowd through an immersive visualization.

D. CyberEmotions: communicating emotions
The CyberEmotions european project aims at supporting the transmission of emotions in the context of social 3Dchatting, i.e. text exchanges with an interface displaying 3D avatars (Fig. 8).This is done through the detection of the expressed emotions in the text sentences [17] and the monitoring of the short and long term emotional states [12].The current perceived emotion is made visible through the display of facial expressions and body language postures.If the intensity is temporarily high, such emphasis can be translated into the display of dedicated animations (e.g.fear in the bottom right image of Fig. 8).
The project is now in an evaluation phase to assess the benefits/drawbacks of this parallel communication channel.Future work will also investigate the impact of transmitting subtle emotions by modulating the symmetry of the facial expressions.

E. VSPACE: Multisensory integration of architectonic space
The use of Virtual Reality technologies in Cognitive Neuroscience has boosted the understanding of bodily selfconsciousness.We now know that basic brain mechanisms supporting aspects of bodily self-consciousness such as first person perspective, spatial location and body recognition can be manipulated using Virtual Reality technologies [18].
The VSPACE project uses different kind of displays -head mounted display, large projection screen, mini-cave-to study the impact of such manipulations in the perception of architectonic space.The result will help identifying the factors that enhance a full experience of space within a virtual environment.

V. CONCLUSION
The Immersive Interaction research group aims to be the place where highly qualified Master and PhD students, together with postdocs meet, brainstorm and synergize on real-time immersive Computer Graphics & Animation, Virtual Reality and Experimental Psychology to make fullbodied interactions in virtual environments as effective as their counterpart in the real world.

Figure 1 .
Figure 1.(a) immersive display with active marker motion capture system, (b) the controlled same-height avatar, (c) controlling a child avatar with the same visual scale as in a and b, (d) egocentric scaling strategy: the virtual environment is scaled as experienced by the child avatar [BMT09].

Figure 3 .
Figure 3. Four screens CAVE: the projection viewpoint is updated with the head location tracked with three active optical markers.

Figure 4 .
Figure 4.The user is seated in the haptic workstation (left); both the arms and the fingers can be guided by force feedback when manipulating an object (right).Physically-based interactions [14] exploit PhysX from Nvidia.

Figure 5 .
Figure 5. Motion capture area used for the study on stepping.Cameras are fixed to the wall, the ceiling or on tripods.On the ground one can see a videoprojected calibrated pattern to guarantee the correct display of targets [15].

Figure 6 .
Figure 6.An example of videoprojected start location (left), the end location is constant in space (middle), measured step sequence (right).

Figure 7 .
Figure 7. (left) reconstructed trajectories viewed from one of the video cameras, (right) synthesized crowd reusing trajectory segments

Figure 8 .
Figure 8. Example of a 3D-chatting scenario involving animated non-verbal communication expressing three emotions, i.e. empathy, anger, and fear.Please note that the text in the top left bubble is typed by the user.