Expression Cloning Based on Anthropometric Proportions and Deformations by Motion of Spherical Inﬂuence Zones

—Virtual tridimensional creatures are active actors in many types of applications nowadays, such as virtual reality, games and computer animation. The virtual actors encountered in those applications are very diverse, but usually have human-like behavior and facial expressions. This paper deals with the mapping of facial expressions between virtual characters, based on anthropometric proportions and geometric manipulations by moving inﬂuence zones. Facial proportions of a base model is used to transfer expressions to any other model with similar global characteristics (if the base model is a human, for instance, the other models need to have two eyes, one nose and one mouth). With this solution, it is possible to insert new virtual characters in real-time applications without having to go through the tedious process of customizing the characters’ emotions.


I. INTRODUCTION
Virtual tridimensional creatures are active actors in many types of applications nowadays, such as virtual reality, games and computer animation.The virtual actors encountered in those applications are very diverse, but usually have human-like behavior and facial expressions.The problem addressed in this paper can be formulated as follows: "Is it possible to define a simple and general mapping strategy of facial expressions between the models of two virtual characters with distinct facial features and mesh topology in real time?" The proposed solution to this problem is illustrated in Figure 1.Facial proportions of a base model is used to transfer expressions to any other model with similar global characteristics (if the base model is a human, for instance, the other models need to have two eyes, one nose and one mouth) using a simple and intuitive deformation system.With this solution, it is possible to insert new virtual characters in real-time applications without having to go through the tedious process of customizing the characters' emotions.
The remaining sections are structured as follows.In Section 2, it is presented the related work found in literature.In Section 3, there is a brief summary of the concepts of anthropometry.In Section 4, it is described the system used for the manipulation and mesh generation of expressions.Section 5 presents the expressions used in the proposed system and case studies.Finally, section 6 contains the conclusions of the work.

II. RELATED WORK
The problem of facial expressions generation is not new.Several techniques for generating expressions and facial animation such as those based on interpolation, parameterization, morphing, muscles, pseudo-muscles, and imagebased deformations are described by Noh and Neuman [12] in a survey.Some of the latest techniques are presented briefly in this section.
Anthropometry techniques have been used to define coding specification of shape and animation of human faces.Some techniques use the anthropometric landmarks to generate models with measure variations.In 1998, DeCarlo et al. [5] devised a system capable of generating facial models for virtual reality applications automatically.That work, which was one of the firsts to use anthropomet-ric techniques [12] for face generation, inspired several other works.DeCarlo used B-Spline surfaces to construct a base model, which respected established anthropometric measures, and perturbed those measures randomly, within predefined ranges, in order to generate other models.Fratarcangeli et al. [6] used the MPEG-4 specification (international standard dealing with animation), that uses anthropmetric landmarks, to define the animation parameters of their technique.Blanz and Vetter [3] treated faces as members of a vector space.Thus, starting from a database of faces, captured through scanning of real subjects, they used Principal Component Analysis (PCA) to determine a set of face basis vectors that could be linearly combined in order to generate new faces within that vector space.They were able to generate realistic models with a great deal of variability.With their morphable model system, it was possible to generate expressions [1], [2] to be transferred to other models.The results were very realistic, but the models need to have the same topology and the expressions need to be captured by 3D scanner.Bui and his coauthors [4] also used morphing techniques for facial animation, where the various expressions need to be generated in advance, for each model.Kahler and his coauthors [7] built a base model consisting of anatomical layers (skin, muscles and bones) associated with anthropometric landmarks.With this system of layers, it is possible to deform muscles, generating different expressions, but for each different model, it is necessary to adjust or rebuild the layers system.Kalra [8] used freeform deformation to generate expressions.The mesh of the face is included in a box containing flexible control points.With the deformation of the box, the mesh of the face is also deformed following the same parameters.Noh and his coauthors [11] used the technique of RBF (Radial Basis Funcions) to generate facial expressions in real time, to be used in videoconferencing systems.They developed a system for facial deformation based on moving points in an influence region.However, the expressions of every person that uses the system need to be generated.Also, the adopted model of influence zones presents some limitations which will be discussed later.In their Expression Cloning work, Noh and Neumann [13] applied similar deformation techniques to transfer different types of expressions between two virtual characters with distinct mesh topology, serving as a base for many other works [10], [14], [16], [17], [18], [19], [20].Some of these works used models with different facial features, similar to Noh and Neumann [13], and some used only human models, but, in order to transfer the expressions it was necessary to find dense correspondences between the models using volume morphing and a cylindrical projection to apply the deformations by RBF.Some other works need to personalize a template model to the input.
All the techniques proposed in the literature use heavy computations, which incurs in high CPU costs.The method presented in this work is based on anthropometry techniques and does not need a step of dense correspondence.It uses landmarks and proportions of anthropometric measurements together with deformation techniques similar to those used by Noh et al. [11], in order to adjust the model through influence zones.That allows automatic generation of expressions of several models (humans, creatures, animals, etc.) after the identification of their facial landmarks.The adaptation system by influence zones also has advantages over the technique used by Noh, which it will be described later.

III. ANTHROPOMETRY
Anthropometry is the science that studies the measures of the human body [15].To perform facial measurements, it uses well-defined points of the face, called Landmarks (Figure 2).The five types of facial measurements (Figure 3) defined with the landmarks shown in Figure 2 are [5], [9]:shortest distance between two landmarks, e.g., ex-en; axial distance between two landmarks, e.g., v-tr; tangential distance between two landmarks, e.g., ch-t; angle between landmarks and one of the coordinate axes, e.g., ear inclination; and angle between points, e.g., mentocervical angle.Measures of human faces from different places of the world have been collected over the past decades, resulting in a database providing the outstanding characteristics of individuals from different ethnicity, age and sex.Through the use of those databases, it is possible to simulate variations in a 3D facial model.First, initial measures are attributed to a 3D model, so that, based on the statistics of measures proportions, the possible variations of dependent measures are analyzed and applied to the model, generating a new model [5].In the present study, first, the anthropometric landmarks are used, to determine measures of a base model with no facial expression (Figure 4).Finally, for each facial expression, it is established the proportion of each measure relative to the corresponding measure of the base model without facial expression (Figure 5).Those proportions are stored to be applied to other models using the techniques of geometric manipulation described in the next section.

IV. MEASURES MANIPULATION
The measures addaptation system proposed in this work was designed with the goal of being simple and generic, and can be used with any triangularized mesh.The manipulation of measures is made by translating spherical influence zones to points that need to be adjusted.This manipulation is described in Equations 1, 2, 3 and 4 and Figure 6, in an algorithm form, and illustrated in Figure 7.The translation is devided in N steps (v/N and t/N) as the equations show, and the algorithm in Figure 6 describes only one of the steps.For each step, the algorithm is applied again to recalculate the new position of the vertices.The algorithm can be understood as follows: the vertices that are closer to the center of the sphere are more influenced and are closely adjusted to the movement of the sphere.Vertices that are farther from the center move very little.The vertices that are outside the influence zone remain undisturbed.In Figure 7, we compare this deformation technique with the deformation caused by the impact of a solid sphere on a spherical object made of modeling mass.In Figure 7A, the deformation caused by the impact has a very close resemblance with the shape of the impacting sphere, but extends away from the contact zone.In figures 7B and 7C, the deformation caused by the spherical influence zone is limited to points strictly inside the influence zone.The deformation is more pronounced as the points are closer to the center of the zone and are exactly zero at the boundary of the zone.The deformation pattern does not have an abrupt change of slope at the boundary and the final effect resembles that of Figure 7A.This creates smoother deformations when the measures are adjusted to generate expressions. where x c is the center of the sphere Figure 6.Influence zones application by translation.By keeping the influence zone partially inside the object to be deformed, effects such as those illustrated in Figure 8 are also possible.The influence zones can be also translated in the opposite direction to the simulation of Figure 7, and start moving with the mesh model partially inside it, as shown in Figure 8.The radius of the influence zones may also vary if one needs to cover a larger or smaller area of the model to be deformed.With different radii, it is possible to achieve very different results with the same translation, as illustrated in Figure 8.Although the concept of influence zones has been used by Noh and his coauthors [11], to deform the models using Radial Basis Functions (RBF), there are important differences in the way these zones are used here.Noh described a system where it is necessary to identify the areas manually (they are fixed regions of the face).To adjust the model, vertices are defined to be moved inside the influence zone, and deform the neighborhood (Figure 9B).The vertices that are outside the zone do not change their position.In addition to the manual work done to identify influence areas for each model and the need of identifying points to be moved in order to adjust the neighborhood, Noh states, in his final discussions, that the system can generate discontinuities if the selected point is moved out of the influence region.Noh also states in his conclusions that to solve these problems it is necessary to define moving influence areas.In the presented system, the influence zones are defined in three-dimensional space by spheres of different radii (Figure 9A).There is no need to set points that will be moved and the sphere does not necessarily have its center in a given vertex of the mesh.In the proposed system, it is the spherical influence zone that moves.The method works as if the center of the influence sphere were a vertex of the mesh.Thus, moving that point causes the position of the influence zone to change instantly like a propagating wave, with no discontinuities.
The system is simple and flexible, producing satisfactory results.It is also possible to use multiple influence zones, if a single sphere does not cover the entire desired area.It is important to note that as the sphere moves, some vertices that are in the boundary of the influence zone can go out of it while vertices that were outside can go inside, as shown in Figure 10.In Figure 10A, the points shown in yellow entered the influence zone as it moved to the position indicated in Figure 10B, and have their position adjusted.Similarly, the points highlighted in green got out of the influence zone after the translation of the sphere.This indicates that in the beginning of the process, they suffered some adjustments, but as they were leaving the influence zone, their new positions do not change any longer.This moving influence zone system eliminates the possibility of discontinuity.
With this mesh deformation system, it is possible to generate a set of facial expressions for the base model (Figure 11), and compute the proportions of the measurements with respect to the expressionless base model's measures.After this, facial expressions can be mapped to other virtual models.That mapping of facial expressions is described in the case study that follows.

V. CASE STUDY
In this section it is described the steps of the expression cloning based on anthropometric proportions and deformations by motion of spherical influence zones Step 1: Definition of the base model.
In this case study, the human virtual model was used as base model in order to define: all the landmarks used in the system (Figure 12) and all the measurements in the neutral face.Figure 13 ilustrates some of those measurements.After this, it is necessary to define the desired facial expressions to be transfered and the deformations rules to be applied.Those rules include the direction and amount of movement of the influence zones and their sizes.
It is important to mention that the radii of the spherical zones are defined by rules based on distance of points.Models that have larger regions to be moved require larger spheres.Thus, the regions to be moved are also defined by measures of the landmarks (Figure 14).After the definition of the landmarks, measures in the neutral face and the spherical radius, it is possible to apply the deformations generating some expressions in the base model, moving the influence zones to the position of each expression (Figure 15).At this point, the proportions of measurements taken with and without facial expressions are stored (Figure 15) to be used later in the other models.Step 2: Definition of the other models that will receive the expessions.
In this case study expressions were applied to models with different geometrical characteristics (human, cartoon, ogre, cat and minotaur) and topologically different meshes (Figure 16).Once the models are selected, it is identified the corresponding landmarks of the new models (Figure 17).In this case study, the process of identifying the landmarks is done by a simple 3D picking system.This is the only step that requires manual work.However, current recognition techniques can be used in order to make the system more automated.In this work, that automation was out of scope, since the focus is on defining automatic mappings between different models' facial expressions.
Step 3: Storage of the neutral face measures of new models.After the identification of the landmarks, the system stores the measures of the new models without expressions.Figure 18 ilustrates some measures in one of the models.Those measures are the same used as example in the human model illustrated in Figure13.Step 4: Application of the expressions in the new models.
Next, the measures of the neutral faces are used, to generate the expressions in the new model, applying the rules of radius definition (Figure 19) and the defined tions used in the base model (Figure 15).Expressions are formed translating spherical influence zones centered in the landmarks (the landmarks do not need to be a vertex of the mesh), causing the deformations of the new mesh (Figure 20).With the proposed system, it is possible to transfer facial expressions for various creatures that have similar global characteristics, such as the cat in Figure 20. Figure 21 ilustrates another expression with deformations in the mouth and the eye regions.Since it is used a human model as a basis, the various models must have two eyes, one nose and one mouth.Despite these global features, the geometries of the models are quite different, as can be observed.Figure 22 ilustrates the base model with some expressions and the proposed expression cloning technique applied to all the models used in this case study generated the results illustrated in figures 23 to 26.

VI. CONCLUSION
This paper presents a system for generation of generic expressions, based on anthropometry and deformation of meshes by spherical moving influence zones.In this study, it  The proposed method ensures automatism, independence of mesh topology and adaptation of models (creatures) with different facial shapes.Other expression cloning techniques need to find dense correspondences between the models using volume morphing and a cylindrical projection to apply the deformations by RBF.Some other works need to personalize a template model to the input.All of these techniques, in this step, requires heavy computations, which incur in high CPU costs.In the proposed method this step is not necessary.Unlike morphing techniques, used to transfer expressions, that requires to generate a database of expressions for each mesh with point-to-point association, the proposed technique uses only one database of proportions, which can be applied to any model with a triangularized mesh, which need not have the same topology.It is not necessary to build structures for each model, unlike the systems with skin, muscle and bone layers, or any other specific intervention for new generated models.Thus, it is possible to use new models in applications that run in real time, with great ease, without the need for greater previous settings.
In the case study it was not used expressions with open mouth.To apply the techniques to those types of expressions it is necessary to define restrictions zones in the mouth.

Figure 1 .
Figure 1.(A) Landmarks of the human model.(B) Base model measures.(C) Generation of expression and storage of proportions of the base model.(D) Expression applied to the second model, using the proportions of the base model.

Figure 4 .
Figure 4. Some anthropometric measurements of the base model.

Figure 5 .
Figure 5. Example of proportion between a measure given by a neutral face and by a face with an expression.

Figure 7 .
Figure 7. (A) Deformation of a modeling mass sphere by a solid sphere.(B) Deformation simulation using the proposed system.(C) In wireframe it can be seen that the deformation does not follow the shape of the sphere, generating smooth adjustments.

Figure 8 .
Figure 8. (A) Deformation in the opposite direction of Figure 7. (B) Deformation using influence zone with larger radius.

Figure 9 .
Figure 9. (A) Proposed system of spherical influence zones.(B) Noh's system of influence zones.

Figure 10 .
Figure 10.Moving influence zones.(A) Yellow points are initially outside the influence zone and green inside.(B) After the translation, the yellow points are inside and the greens are outside the sphere.

Figure 11 .
Figure 11.Generation of facial expressions by spherical influence zones.

Figure 12 .
Figure 12.Base model and the landmarks used in this case study.

Figure 13 .
Figure 13.Some measurements in the neutral face.

Figure 14 .
Figure 14.Definition of the radius of some spherical influence zones.

Figure 15 .
Figure 15.Example of aplication of deformations and storage of measures proportions.

Figure 16 .
Figure 16.Models used in the case study.

Figure 17 .
Figure 17.Identification of anthropometric landmarks in the new models.

Figure 18 .
Figure 18.Measures of the new model without expressions.

Figure 19 .
Figure 19.Application of radius definition rules.

Figure 20 .
Figure 20.Example of proportion of the base model and the expression transferred to the new model.

Figure 21 .
Figure 21.Application of proportions in a cat model.

Figure 22 .
Figure 22.Expressions applied in the base model.

Figure 23 .
Figure 23.Expressions transferred to a toon model.

Figure 24 .
Figure 24.Expressions transferred to a cat model.

Figure 25 .
Figure 25.Expressions transferred to a minotaur model.

Figure 26 .
Figure 26.Expressions transferred to an ogre model.