MamuLEDs: Mixed Reality meets Mamulengo

Techniques of Virtual, Augmented, and Mixed Reality (VR, AR, MR) have been used for puppet theater in different cultural contexts around the world. However, we are not yet aware of the use of these techniques in the Brazilian Northeast Popular Puppet Theater, a tradition known in Pernambuco as Mamulengo . We present here a system developed for “The Quarrel Between the Real Puppet and the Virtual Puppet,” a sketch created for the class Realidade Virtual e Aumentada of Computer Science program at CIn-UFPE. This paper aims to investigate how VR/AR/MR techniques can contribute to the creation of mamulengo shows. We used the method of qualitative case study, analyzing the audiovisual records of essays and presentation, the software solution developed in Unity 3D, and the data collected from a focus group interview with puppeteers. This study identified as weaknesses of our system: the hand tracking unreliability and the lack of tactile feedback, causing discomfort in experienced puppeteering professionals. As strengthens of the system, we found that the technology aroused interest because of its potential for expanding the options of tools available for popular puppetry shows creation in current times.


Introduction
In this paper, we present a qualitative case study about the computational system used in "A Peleja do Boneco Real com o Boneco Virtual" (The Quarrel Between the Real Puppet and the Virtual Puppet) 1 , a puppetry sketch using Mixed Reality technologies, like hand pose tracking.
Puppetry is present all over the world and took on different aspects and dramatic spirit, due to the cultural traditions in each geographical location. The Teatro de Bonecos Popular do Nordeste (Northeast Popular Puppetry) was officially recognized by IPHAN as a cultural heritage of Brazil in 2015 2 . In the State of Pernambuco, the term "mamulengo" refers specifically to the glove puppetry tradition; the puppet head is sculpted in wood (and other materials), and its body, made of cloth, which the puppeteer wears like a glove. 1 https://youtu.be/KfUlb587m18 2 http://portal.iphan.gov.br/pagina/detalhes/508/ Puppetry has props, scenery, and tent as physical elements in addition to the puppets. The fable narrative atmosphere allows the creation of an enchanting environment in the interaction with the audience. Traditionally, its realization does not depend on digital technological devices.
However, the cultural manifestations of tradition, in the dynamic process within a social environment, can incorporate new expressive resources in its historical trajectory.
There is something that is not extinguished by the novelties of the modern world, its technologies and social changes, that tradition and modernity are related, dialogue, negotiate among themselves ways of re-elaborating symbolic values that can be understood and accepted by current individuals (Macêdo & Assunção, 2015). From left to right: Fernando Alvim with Justino Guerreiro (Real Puppet), Ricardo Rossiter (Virtual Puppet), Jarbas "Jabah Pureza" Jácome (Guitar), Denyse Gonzalez with Dandara (Niece of Real Puppet and Narrator 1), Kennedy Albuquerque (Synthesizers), Maria Oliveira (Narrator 2) and Mavi Pugliesi (Pife).
As occurs in many parts of the world, we can try to use these digital technologies as a support tool for puppetry, expanding its aesthetic horizon, if the mamulengueiro 1 so desiresdepending on their need to expand communication possibilities.
The system presented here uses real-time hand pose tracking that allows the control of a virtual puppet projected on the stage. Figure 1 is a photo of the sketch's open essay and shows that there is an interaction between the real environment of the puppets and another 3D environment projected behind, where is the so-called "Virtual Puppet." Therefore, we consider an example of Mixed Reality following the definition of MR as an "alignment of environments," described in "What is Mixed Reality?", definition 5 (Speicher, Hall, & Nebeling, 2019).

Related work
We performed an automatic search to find previous studies applying concepts of Augmented and Virtual Reality in puppetry. The search was carried out on the Scopus 2 database, using the following string: puppet AND ("virtual reality" or "augmented reality"). About 400 results were returned, of which 11 were selected. We present below those researches that we believe have conceptual or technical proximity to the case we analyzed in this paper. We will follow a chronological order, as it shows the changes over time in hand tracking systems and graphic machines used.
The research "Computer Animation and Virtual Reality for Live Art Performance" presents solutions with few possibilities for articulating the puppet, whose movement needs to be complemented by procedural animations (Jessel, Jaspart, & Flores, 2001). The system presented uses Polhemus magnetic tracking sensors on the hands and other parts of the manipulator's body. This technique makes it possible to control members, eyes, nose, and mouth directly by associating them with the values obtained by the sensors or by interpreting these values as changing the axis of rotation or changing the range of motion. The research was carried out with experienced puppeteers.
The study "Virtual marionettes: A system and paradigm for real-time 3D animation" describes the simulation of puppets controlled by wires using a 3D physics machine developed by the authors, with collision detection, considering the parts of the puppet as a set of particles instead of a rigid body (Bar-Lev, Bruckstein, & Elber, 2005). The system uses a simplified "data glove" that obtains a value corresponding to the articulation of each of the five fingers of the hand. The system allows the user to assemble the puppet with 3D parts provided. It also allows the association of the operator's 1 Mamulengueiro is the word used in Northeastern Brazil to nominate the people who construct and animates the mamulengo puppets, creates the plays and performs with the puppets. By its time, mamulengo designates, in Pernambuco, both the show and the puppet. The word mamulengo, according to Gurgel, (1986, p. 11) and Borba voice amplitude to the movements of the mouth of each puppet. In the conclusions, the article mentions that there were no tests of the system by professional puppeteers.
The research "Building Hand Motion-Based Character Animation: The Case of Puppetry", presents a system for simulating puppets of the same type that we studied in our case: the so-called glove puppets (Luo, Chen, Yeo, Lin, & Li, 2010). Unlike marionettes, movements of the glove puppets are determined by the conformation of the fabric and upper limbs of the puppet to the hand pose changes performed by the manipulator. The researchers themselves developed both the hand movement capture device (called "SmartGlove") and the 3D animation software (called "IM-Puppet"). SmartGlove uses optical linear encoders to obtain finger positions. The arms and head position is determined in real-time by fingers position on the SmartGlove. Also, the system allows pre-programmed procedural animations such as walking. The paper mentions the participation of professional puppeteers for the creation of procedural walking animation but does not report tests of evaluation of the system by these professionals.
The play "Pictures at an Exhibition" uses puppets built with sensors that, when moved by manipulators, trigger projected animations of abstract images (Mazalek et al., 2011). An abstract painting inspired by a Kandinsky is animated according to the movements of the puppets in real-time.
Considering the set of studies that we are presenting, this is the first that uses Unity3D for real-time animations. The paper reports the intense collaboration with professional puppeteers in the research.
In our literature review, the research "Character motion control interface with hand manipulation inspired by puppet mechanism" is the first using the Leap Motion sensor for hand tracking in order to manipulate virtual puppets (Oshita, Senju, & Morishige, 2013). The researchers used the sensor to control a marionette, mapping different points of the hands to different simulated strings of the virtual puppet. The paper reports that there was no evaluation of the system by professional puppeteers.
The study "Puppet Narrator" evaluates the use of virtual puppet control programmed in Unity3D, via Leap Motion, for children storytelling (Liang, Chang, Kazmi, Zhang, & Jiao, 2015). The system associates some gestures with preprogrammed movements of the virtual puppet, without directly associating finger movements with puppets members. The study does not report evaluation by puppet makers.
The researcher Luis Leite has several publications in the area of interactive computer systems applied to the puppetry. His work "Mani-Pull-Action" reports the development of a system using Leap Motion as an input for controlling marionette-style virtual puppets, developed in Unity3D (Leite & Orvalho, 2017). Each finger is associated with a Filho (1966, p. 86-87), probably come from "mão molenga" ("floppy hand"), a reference to the flexibility of the puppeteer's hands, needed to give expression and liveliness to the puppet. type of movement of the virtual puppet, such as eyes, mouth, nose. Moreover, the rotation of the hand's palm is associated with the rotation of the puppet's body. The author reports Leap Motion tracking unreliability problems, but presents positive results evaluation in research with dozens of participants. Non-specialists are the target of this work. However, it reports an intense collaboration with professional puppeteers.
The research "Unreal Interactive Puppet Game Development Using Leap Motion" uses Leap Motion as an input for a fighting game in which each hand tracked by the sensor controls a puppet (Huang, Huang, & Jhu, 2018). Puppet rotation and position are associated with the player's hand rotation and position. The flexion of each finger is associated with a character's fighting move. The researchers used Unreal as a development platform. There was no evaluation by puppeteers because the objective was to develop a game and not a platform for puppetry.
In "Virtual Stage: Interactive Puppeteering in Mixed Reality", the researchers used Leap Motion to recognize simple gestures (Husinsky & Bruckner, 2018). There is no direct control of puppet members. In this work, the puppet handler uses an HMD (head-mounted display), allowing greater immersion in the virtual environment. The system allows the automatic association of the puppet's voice amplitude variation to mouth movements. The paper does not report evaluation or work together with professional puppeteers.
The research "Combining leap motion with unity for Virtual Glove Puppets" had as motivation to disseminate the tradition of Chinese glove puppets (Lin et al., 2019). The 1 https://vvvv.org/ system was developed in Unity3D using Leap Motion for hand tracking. The palm is associated to the puppet's body, the thumb to the left hand, the little finger to the right hand and the middle finger to the head. This paper emphasizes the potential benefits of using VR/AR/MR technologies to strengthen of the puppetry tradition in Taiwan.
Presented at VISIGRAPP 2020, the "Expanded Virtual Puppeteering" (Velho & Lupiac, 2020) is another Brazilian research using Leap Motion and Unity3D for virtual puppetry. The system recognizes a set of pre-defined gestures, pre-programmed puppets movements, and different modes of puppet control using the manipulator's hand pose tracking. It allows the use of HMDs, expanding the possibilities of show formats. "A collaboration with a professional puppeteer and a play director was made." Table 1 lists the presented studies and shows the change over the years of hand tracking systems and graphics engines for 3D applications with the development of Unity and Unreal.

Method
We adopted the Qualitative Case Study method as proposed by Stake (2005) for research that seeks to construct knowledge from the relationships of a case and its context. The author suggests three overlapping categories case studies: (1) of intrinsic interest, (2) of instrumental interest, and (3) collective (or multiple) case studies. Studies of intrinsic interest are those of which the main objective is to understand the case itself, in its uniqueness and relationships with Leap Motion + HMD Unity3D Yes contexts. In essence, the case, and not a general problem, is the starting point of the study. Studies closer to the instrumental interest category start from a general problem and analyze a case, contributing to the understanding of that problem, by observing the functioning of the case in its context.
The research presented here is closer to a case study of instrumental interest because it starts from the general problem that can be expressed by the research question: how can VR/RA/RM techniques contribute to the creation of mamulengo shows?
Stake suggests the following steps for a qualitative case study.
Bounding the case, conceptualizing the object of study. Our case consists of the computer system developed for the sketch comedy "A Peleja do Boneco Real com Boneco Virtual," detailed in section 5 Case description. Information relevant to the case analysis and comprehension, but which is not part of the case itself, is presented in section 4 Contexts of our case.
Selecting the phenomena, themes or issues. Guided by the research question, we focus on finding out what contributed in terms of software and hardware technology decisions to the process of creating, testing, and presenting the referred comedy sketch.
Seeking data patterns to develop the issues. The data analyzed for pattern searches were collected from the discussions held during the focus group in which we discussed the positive and negative points of the developed system and the sketch comedy process itself. Other sources of data were used to search for patterns: video and audio records of rehearsals, meetings, and playing.
Triangulating key observations and bases for interpretation. We identified the relationships, correspondences, and divergences between events observed in the rehearsal and presentation videos, the software design decisions, the video recorded system evaluation with comments made by our mamulengo specialists during the focus group.
Selecting alternative interpretations to proceed. We selected the interpretations that we found more relevant to the research question.
Developing assertions or generalizations about the case. We present the assertions constructed in section 6 Evaluation and discussion. This case study did not aim to find generalizations about the problem.

The puppet and its essence
The puppet is the main element within a puppetry show. It is the one who relates directly to the public. Its symbolic plasticity and movements generate impacts on the audience in order to produce immersion within the fictional reality created. According to Amaral (1996, p. 296), on the stage, isolated from its environment, under the scenic lights, with the addition of movements, animated, the object acquires a force that goes beyond its functions and its matter.
The puppet within the animation game is not just a simple object. It is not just a matter of transferring the animator's movement to an object. It, the puppet, through movements, comes to life. Animating a puppet is not just about making random hand movements. The action needs to be convincing. There is not just the pure desire of the manipulator to gesticulate with the puppet. A gesture is combined with interpretation, emotion. The puppet, therefore, is used to represent the dramas of everyday life and to approach themes and put them in a discussion.
The manipulation of the glove puppet happens through the introduction of the puppeteer's hands into the puppet's clothes, setting up the body. The movement of the puppet's head and arms occurs through the movements of the puppeteer's fingers. The puppet is, therefore, an extension of the body of the puppeteer, without any mediation.
The actor-animator, actor-puppeteer, seeks to produce the impression of life in a body that seems external to his own body, but which depends exclusively on his body. It is necessary a gesture palette that will always be linked to the actions of the characters and their relationship with the story. To interpret a puppet is to give it a body, a voice, and all the gestures that give it action and life. The actor-animator and his puppet form an organic whole, and neither can be considered more important than the other.
There are sudden, smooth, exaggerated, subtle movements. Everything has its intention. In the game of makebelieve, it is necessary to define the sequence of movements so that the technique is associated with emotion, avoiding "mechanism." It is an arduous task of setting up the characterization of the puppet, in the work's context as a whole. The actor-puppeteer needs to be mindful of the scenic direction so that everything is coherent with the proposed script dynamics. Beltrame (2003, p. 22) states that it is possible to consider the puppetry as a language with its own rules, which are in the continual process of transformation, which can be updated, recreated, or overcome. The most recent events in the different fields of the arts reveal changes, showing a movement towards the expansion of the forms of performance, which are mixed with other artistic languages.
The artistic process of Mamulengo occurs within an artcraft system, an age-old tradition. Players are symbiotically linked to the puppet and other elements that make it up (tent, props, scenery, and costumes) with all their playfulness and skills developed through training, learning, and exchange of knowledge. This environment of creation, for the most part, takes place inside the puppets' own homes, where the family participates in these inventive moments. The elements that are part of a show take shape from conversations and daily observations.
Mamulengo is a game that preserves its poetic-theatrical structure, of which performances are irreverent and full of improvisation, following simple scripts created by the puppeteers themselves. It is a continuous binding between fixed form and inventiveness. Acting masterfully within their social realities, we have to consider the changes within their own lifetimes' route. Tradition indeed seeks to preserve the game with all its creative simplicity. It is also true that a puppeteer is always attentive to innovations and introduces elements that raise a curiosity in its public presentations.
It is common to find a puppet made of plastic or other material within a show, among the ones made of wood. This odd figure has a prominence among the others, being able to cause laughter and discussions. Meschke (1974) states that: it is not a crime to combine puppets with live actors, to use puppets with live actors, to use very large or tiny puppets, to combine puppets with masks, dance, opera, drama, cinema, music-hall, pop-music. To use puppets in two or three dimensions, or both together, to mix varied puppet techniques in one production. To replicate puppets in smaller versions for perspective effect. To appreciate Punch & Judy, Kasper, Guignol, or whatever name they have. To let the manipulator and his technique visible or not during the show. To work for months to achieve the aesthetic perfection of a puppet or to use a match as a puppet.
Paper-mache, for example, is a mass made of shredded paper. It is an art developed in China, around two centuries before Christ. In Europe, it was used to create decorative objects and was introduced in Brazil since the arrival of Europeans. This technique presents itself as an adequate alternative for raw material because it is a low-cost and easy-tomodel material, in addition to strengthening the concepts of recycling and reusing paper (Perez & Yamada, 2015).
In the years of 1960, with more puppetry use in education, the puppet made of paper-mache has strengthened and today remains at the artistic environment alongside with puppets made by wood, sawdust, polyethylene, and many other materials. On the other hand, many puppet masters with more traditional thinking will claim that the "real mamulengo puppet" is made of wood. However, this does not forbid them from dealing with other materials. Nowadays, many of these masters offer workshops about the construction of puppets with objects subject to reuse and also teaching the secrets of carving wooden puppetsthe apples of their eyes. As Castro (2015) states the puppets are traditionally made of wood, of which are sculpted their heads and hands, and also of cloth to make the dressing. The dramatic structure of the show calls for the puppet to be tough and durable. The option for making it with wood may be in the concern with the durability of the material, after all, they cannot be fragile puppets, since, in a single action, they might be hitten by a club, they might fall.
Mamulengo needs an attitude, a way to deal with the puppet as a representation of the human being. Mamulengo is a whole set of knowledge and playfulness. Masters of mamulengo are players that, in their spontaneity, often seem children. Their contribution to our culture involves music, plasticity, and orality, constructing knowledge transmitted from generation to generation. Mamulengueiros are players open to challenges, despite agreeing with the playing traditional rulessome more than others reinvent themselves and discover new paths.
Since the late 1960's, Hermilo Borba Filho drew attention to this movement. Popular artists incorporate and absorb any new everyday occurrence that hits their imagination, a fact that does not represent corruption in their art. Influences succeed, but the most important thing is to maintain the popular spirit (Borba Filho, 1966).
Mamulengo is one of many popular traditions celebrated in the Northeast region of Brazil and specifically in the State of Pernambuco. This cultural atmosphere in Pernambuco is combined with technology enthusiasm motivated by institutions like the CIn-UFPE 1 and research groups like MusTIC 2 and Voxar Labs 3 . This combination produces a fertile environment for creations of new art experiments applying the Computer Science achievements to popular traditions. In the following sections we present one example of this kind of experiment: a puppetry sketch involving many artists, designers, and computer scientists created for a final project of a Computer Science class at CIn-UFPE.

The Quarrel Between the Real Puppet and the Virtual Puppet
The text of the sketch "A Peleja do Boneco Real com o Boneco Virtual" is inspired by cordel, which is a centuriesold Brazilian tradition of street literature with part of its origins in medieval Portuguese popular prints, ballads, chapbooks. The peleja is a common form of cordel. Considering its use in the context of cordel literature, Slater (1982, p. 55) translates this old Portuguese word "peleja" to "poetic contests." Cordel pelejas draw heavily upon the oral tradition. Despite the impression created by early folklorists such as Leonardo Mota, however, these folhetos [another noun for cordel] are literary compositions rather than transcriptions of actual oral performances. In them, two more or less evenly matched competitors seek to outdo each other in questions of objective knowledge and verbal skill (Slater, 1982, p. 65).
Indeed, the orality of cordel pelejas writings resembles another centuries-old tradition of sung poetry, also still strong nowadays in the Northeast of Brazil, called repente or cantoria (Sautchuk, 2011). It is common to see in repente the two poets (called repentistas) taking on a poetic contest, disputing who is the most virtuous, most of the time within a friendly and comic atmosphere. They improvise sung verses in specific melodic structures accompanied by their guitars. The repentistas improvise verses with rhymes for a cheering audience. Therefore, it is possible to find some similarities with the Hip Hop "battle rap" practice. "MCs will perform on the same stage to see who has the better verses" (Edwards, 2009, p. 27).
In our sketch, the peleja or poetic contest is between a character called Boneco Real (Real Puppet) and another called Boneco Virtual (Virtual Puppet). A glove puppet inspired by the Mamulengo tradition, made with paper-mache and cloth, represents the Real Puppet. The Virtual Puppet is represented by a digital 3D modeled version of the Real Puppet design, exhibited in the scene by a digital display. Two other elements of the Virtual Puppet are intentionally visible as part of the scenery: the sensor used for hand pose tracking (Leap Motion) and the "nude" hand of the actormanipulator. A third character is the Sobrinha do Boneco Real (Real Puppet's Nice) represented by another traditional style paper-mache glove puppet.

Spot
The Virtual Puppet observes the Real Puppet and provokes him, calling him "tropicalist garbage." The Real Puppet responds by expressing pride in its condition of a popular Brazilian tradition representative. The Virtual Puppet replies with arrogance, boasting an alleged superiority and technological sophistication. The Real Puppet points to the disadvantages of its opponent's digital technological nature, such as the lack of tactile materiality, dependence on electricity to exist, and the presence of "bugs" like any other software.
The dispute is becoming increasingly tense. The Real Puppet accuses the Virtual Puppet of cultural appropriation to gain artistic recognition and therefore owes the Real Puppet the royalties for using that body image. The Virtual Puppet answers that the 3D model avatar can be easily replaced by any other. The Real Puppet then reveals that his niece had already patented the hand pose tracking technology used by the Virtual Puppet, and the two put themselves on the verge of physical aggression.
The Niece of Real Puppet suddenly enters the scene. She interrupts the quarrel between the two characters by proposing a commercial agreement between them so that the wealth of the Virtual Puppet is fairly distributed to mamulengo masters of Brazil.

The process of creation
We present here the sequence of events that occurred since when the idea of the sketch came about, going through the development process, essays, presentation for evaluation as the final project for a Virtual and Augmented Reality class. The narrative goes until the internal evaluation of the system through a focus group interview with the team members experienced in mamulengos.
This context is presented chronologically, organized by dates in the form of a narrative, which is an adequate form of presentation for case contexts in qualitative case studies (Stake, 2005, p. 454).

Case description
Our case is the computer system that enables the existence and functioning of the Virtual Puppet. Its main functionality is controlling a 3D puppet according to the movements of the hands and fingers of a manipulator. These movements are captured in real-time by a hand pose tracking system. Like most of the related works presented in this paper, we use the Unity3D game engine and the Leap Motion sensor. Inspired by the tradition of mamulengo glove puppets, we chose to use the index finger to control the movements of the head and the thumb and middle fingers to control the arms. Also, the orientation of the palm controls the orientation of the puppet's body (Figure 8). We associate the element of the hand that corresponds to the desired joint for each member of the puppet to be articulated. This association is made through a C# script that copies the global rotation value of the hand element to the local rotation of the puppet member. The fact of associating the global value of the hand with a local value on the puppet causes the movement on the puppet to be amplified and, at the same time, distorted. This distortion was maintained for aesthetic reasons and also because when we turn the hand to the left, the puppet's head turns at an angle twice as large, as it is applying the global rotation of the hand as being local in the finger. As the head's GameObject is "son" of the body GameObject, and the body is being rotated, the head rotates one time more than the body. This effect of turning head on the body is also possible to be achieved by traditional glove puppet's experienced professionals.
When adding the C# script as a Component in the GameObject of the right arm, for example, we can add the reference of any GameObject in the "Controller" variable. The GameObject, called "L_middle_a," and corresponding to the beginning of the middle finger, was added via the Unity3D graphical user interface as the GameObject to be read (Figure 9). The integration that already exists between Leap Motion and Unity3D through the Interaction Engine 1 , as well as the modular and flexible graphical user interface of Unity3D, allowed the development of a prototype in a few minutes and with practically just one line of code. Other features have been added to the software, such as an animation of an analog TV turning off and other simple effects triggered via a simple state machine controlled by a keyboard. In our sketch presentation, the guitarist used a modified computer keyboard 2 as a kind of DIY pedalboard to be able to control the Boneco Virtual's stage effects using his feet while playing the guitar.

Evaluation and discussion
Here we present the results of the theme selection process, search for patterns in the data, triangulations, interpretations, and construction of assertions following from our case study.
We will take as a starting point to present these results, the focus group interview held at the Escola Municipal de Arte João Pernambuco on July 12, 2019, in which Maria Oliveira, Fernando Alvim and Jarbas Jácome were present.
The objective of the interview was to evaluate and discuss the possibilities of using VR/AR/MR technologies, specifically the hand pose tracking system, in the context of Mamulengo tradition. We justify the need for the presence of each of the three participants below:  Maria Oliveira, has 38 years of experience in the Popular Puppetry of the Northeast, during which she developed a professional relationship with several masters of this art.  Fernando Alvim was a student of Maria Oliveira in her recent "Puppet Course: Making and Handling," exactly where he conceived and made the Justino Guerreiro puppet.  Jarbas Jácome created the first version of the sketch text, implemented the system software in Unity3D, and could act as something close to a moderator during the interview.
Despite having participated in the creation of the sketch presented in this paper, neither Maria Oliveira nor Fernando Alvim had the opportunity to try the Virtual Puppet hand tracking technology before the focus group.
The interview lasted two hours, and it was structured to follow an alternation of technology experimentation by the participants, opening up for debates between the turn of one and the other. Nevertheless, it ended up following a less structured dynamic because of a technical problem, that is detailed in the last subsection of this evaluation. Participants tried the technology for a few minutes while we debated. The excerpts from the transcripted dialogues originaly in Portugueese are translated to English here. The translation was reviewed by each participant. We can organize the issues raised in the following topics.

Unreliability of hand pose tracking
Some studies presented in section 2 Related work inform problems with Leap Motion handtracking. Indeed, despite its high precision (Weichert, Bachmann, Rudak, & Fisseler, 2013), Leap Motion has hand pose tracking reliability problems that have been reported since the time it was launch. "It cannot currently be used as a professional tracking system" (Guna, Jakus, Pogacnik, Tomazic, & Sodnik, 2014). Sunlight presence in the environment accentuates this problem because of infrared interference (Figure 10). Fernando Alvim described this issue using the expression "involuntary movements [of the virtual puppet]." Maria Oliveira, at one point, stated: "We were afraid [of the virtual puppet] to lose his balance." The problem also occurred in our open rehearsal, the final presentation for the RVA class, mainly at the beginning, as can be seen in the first moments of the video when the Virtual Puppet enters the scene 1 . We believe this is related to the entry of indirect sunlight into the room. However, in our case, the reliability increased during the presentation, which was not compromised.
As a result of the study on this topic, we arrived at the following assertion: Assertion 1 -For use in professional shows, the system needs to be tested beforehand with the ambient lighting condition as similar as possible to the one that will exist in the moment of presentation.

Sensor reduces freedom of puppeteer's body movements
"With the virtual puppet, I do not move my entire body. I am manipulating, but I am on edge, [with] a limited focus. With the real, we have more freedom. The puppet is an extension of my body, manipulating it is a dance." Said Maria Oliveira.
The sensor's setup that we used both in the sketch presented, and in the focus group interview, was with Leap Motion positioned still, pointed to up. This configuration forces the puppeteer to move within a limited space compared to the freedom one has when using a traditional glove puppet.
Assertion 2 -With the motion sensor stood, instead of glued to the body, the freedom of movement is drastically reduced.

Impact of tactlessness on the puppeteer
During the focus group interview, the most frequent issue pointed by Maria Oliveira was the absence of touch in the hand tracking system for puppet control. "It is not mamulengo. It is another manipulation logic. Mamulengo is visceral: the puppet is the extension of my body. [With the virtual puppet], I do not feel the weight, the volume, the touch is just pretending." 1 https://youtu.be/KfUlb587m18?t=50 The related works Table 1 shows that the latest experiments in our literature review did not use the gloves approach as sensors for detecting the movement of puppeteers' hands. In this case, what a computer scientist or a designer may see as an evolution, a puppeteer can see as a step backward, given the emotional importance of physical touch in this activity. These discussions led us to the following assertion: Assertion 3 -The absence of tact is a factor that conceptually distances the Virtual Puppet system from its primary inspiration: Mamulengo tradition.

Aesthetics, technology and tradition
Returning to the question of "viscerality" raised by Maria Oliveira, cited in last subsection, we propose now the following imaginative exercise: what would it be to pull the viscera out of a mamulengo puppet? According to a literal sense of the term "viscera," we could answer by plucking "the organs located in the thoracic and abdominal cavities" of the puppet's body. That is, it would be to pluck the puppeteer's own hands from inside the mamulengo puppet glove.
Developing this image, we can perceive the brutality that we commit with the Mamulengo Art by exhibiting a floating "puppet viscera" on the stage, controlling an external body image appearing in a digital display.
Maria refers to tracking as a "motion-sensitive technology". "I do not have the puppet. I just see it! It does not interact with my body. It looks like a robot. There is a relationship of strangeness in this visual manipulation". Therefore, it seems to be evident the fact that the goal of the technology is to sense. It (the sensor technological object) reveals its purpose very quickly. Because of this cold straightness, the technology does not play with the puppeteer, like the traditional one. The traditional puppet does not just want to sense; he wants to resist too. The puppet is stubborn. Leap Motion is an eye. It senses the hand movements, but the "living" part is distant. The hand is nude, like the dramatic viscera of an imaginable puppet, floating in the air, without the rest of the puppet. The hand thus feels cold; the mamulengo puppet, differently, warms the puppeteer's hand. Even so, Maria Oliveira says: "[The Virtual Puppet] is sympathetic." And Fernando Alvim says: "I like this technology for the novelty. It challenges us to think about Art within new possibilities".
Following this discussion, Fernando pointed to the metafictional aspect of the sketch's text and scene composition. The cordel literature form of the pelejas (poetic contests) chosen by Jarbas is used to denounce the superficiality of the supposed dichotomy between the so-called "new digital technologies" and the "popular traditions." We can mention other elements that can contribute to this reading, such as the organization of the scene evidencing the sensor, the nude hand without a puppet glove covering it, the dialogue about the proper functioning or malfunction of the Virtual Puppet despite this character boasting about its technological so-phistication. These elements can contribute to the construction of a critique and complexification of what we defend to be an unproductive dichotomy between technology and tradition, through humor. These elements try to ridicule the passionate defense of one of the two extremes. On the one hand, we have a pretensious crystallization of tradition as something pure and frozen in time. On the other, we have a worship of technological novelties whose operating mechanism knowledge are egoistically closed and which products are fetishized by industrial competition and marketing.
As we read in the subsection 4.1 The puppet and its essence, the metafictional approach to include industrial novelties is not rare in Mamulengo Art. An example given is the puppet made of plastic, considered odd, causing laughter and discussion between other puppets made with traditional materials.
These aesthetical deviations, strangeness and consonance with prior traditional Mamulengo practices led us to the following assertion: Assertion 4 -Despite being conceptually distant from traditional Mamulengo Art, Virtual Puppet technology can contribute to the discovery of new artistic possibilities, from the interaction with traditional modalities of popular culture.

Virtual Puppet's development tool
The exchange of knowledge between masters is part of the culture of Mamulengo. This exchange takes place either in informal meetings or in institutional training spaces, through workshops or courses.
Throughout our sketch creation process, members of the group raised the possibility of sharing the technical knowledge for the Virtual Puppet's computer system development. When this question was raised, Jarbas Jácome replied that a Unity3D and Leap Motion workshop could be organized to teach Virtual Puppets development, considering, however, the existence of a barrier for non-programmers without experience with Unity3D. However, due to the speed with which the prototype was produced, Jarbas mentioned that for someone with some minimal experience as a software user, small modifications and adjustments to the system would be trivial activities and easy to learn in short workshops.
However, an unexpected technical difficulty occurred during the focus group interview that changed that perception. Early in the experimentation session with hand pose tracking technology, Maria Oliveira and Fernando Alvim questioned the fact that the system worked only with the left hand. Maria pointed this out as a problem for professional use since the technique is polished during years of practice and, therefore, the preference for using the left or right hand might be critical, depending on the practice of each puppeteer. Jarbas Jácome then asked for a few minutes to fix this problem, judging it to be trivial to do, quickly modifying the software programming in Unity3D to work with the right hand instead of the left hand, since both Maria and Fernando are right-handed. Nevertheless, after about one hour of focus group interruption for that, Jarbas finally gave up, as he was unable to get the Virtual Puppet's rotation to work correctly with the right hand. He thought it would be enough to change the hand provided by the Leap Motion Interaction Engine associated with the puppet's GameObject. However, no matter how the values, signs and rotation settings were changed, the puppet did not work correctly with the right hand, and it was not possible to identify what was wrong.
This event led us to question the idea previously defended by Jarbas in rehearsals and meetings with the team, that the system created is extremely simple and easy to be programmed and personalized by someone with little experience in Unity3D. This impression was given by the fact that the prototype was developed in just a few minutes and using a few lines of code. However, the rotation adjustments made for the left hand during the programming of the prototype, through trial and error, in a few minutes, could not be easily adapted to the right hand at a practical time during the focus group.
From this event, the discussion raised during the focus group and evaluating the system's development iterations we arrived at the following assertion regarding the theme of the Virtual Puppet's development tool: Assertion 5 -Despite facilitating extreme rapid prototyping for a specific case, Unity3D did not guarantee ease for quick modifications pointing to the need for a specific editing tool applied to the puppetry domain, with user-centered design and usability assessment.

Latest developments and future work
In addition to some possibilities for future work, in this section, we will present a consequent international development of the experience presented in this case study that occurred after the finalization of the first version of this paper (in Portuguese) published and presented in SVR2019 (Jacome, Oliveira, Alvim, Teichrieb, & Ramalho, 2019).
In the second semester of 2019, we held a workshop in California, applying the idea of the Virtual Puppet using the free software Pure Data and the GEM (Graphics Environment for Multimedia) extension, replacing the proprietary Unity3D software. The invitation came from an email conversation including computer scientist Joseph Deken, founder of New Blankets 1 , during the construction process of the event PdWeekend 2019 of the Pure Data community in San Diego and Los Angeles.
During the email conversation, months before the event, our technological experimentation involving Mamulengo tradition drew the attention, especially of the Dutch artist Katja Vetter, a specialist in performative sound art. Katja and Jarbas were invited for PdWeekend2019 as "Artists-in-Transit fellows." New Blankets supports the travel and activities for the Artists-in-Transit as unpaid volunteer experts.
books, technology, training, and human experts for free/open diffusion." We started discussing by email, and some video calls between Brazil, the United States, and the Netherlands, possible formats for a workshop about digital puppets models controlled by hand pose tracking using free software.
Katja suggested that, before arriving in the US, we would focus on the development of two independent, yet complementary systems: one for controlling the Virtual Puppet using hand pose tracking and 3D rendering and the other for real-time voice processing.
For the first, we made a new version of the Virtual Puppet system using Pure Data for communication via TCP/IP with a C++ code to obtain Leap Motion data (following the suggestions by Miller Puckette, sent by Katja) and GEM, Pd OpenGL extension, for rendering the puppet. We choose to use Pd/GEM instead of Unity3D in order to have the opportunity to experience another development tool, made by a community traditionally interested in music, performartive and interactive art. Compared to Unity3D, Pd/GEM is a lightweight software with more advanced native real-time sound processing features given by Pure Data. For the second system, Katja took advantage of some parts of the software she develops in Pure Data and uses in her sound performances. Thus, Personatrix 1 emerged: "an effects processor, providing sonic tricks to create imaginary characters." After months of conversations and local articulations, Joseph Deken confirmed that the workshop would be for the first graders of HTe -High Tech Elementary, which is part of High Tech High, a San Diego, California-based network of charter schools. The Teacher Claire Deken made this possible.
On September 29, Jarbas Jácome took to California, sixteen mamulengo puppets, made by Maria Oliveira especially to be demonstrated and eventually sold during the event. Some of these puppets were loaned to Claire Deken's classroom. She had the idea to present them to HTe students during the week as a way to bring them up to the theme of international puppetry, some days before the workshop.
We elaborated on some possibilities of activities that could be carried out with HTe students. However, the decision of what the meeting would look like was only taken after we got to know the school environment better. We followed an approach inspired by Paulo Freire's ideas, who proposes to know the reality of the students before elaborating on the content to be worked on in the classes. Joseph Deken is a thinker who applied Paulo Freire's ideas to New Blankets workshops for many years.
On September 30, we visited the HTe school to understand the environment, learn a little about the students' experiences by talking to Teacher Claire Deken and discovering contents of interest to the students. She showed us the students' preferred story: "I'm a Frog!" by illustrator and writer Mo Willems 2 . We decided to use this story as the content for our puppetry presentation. After that first visit to the school, Jarbas modeled the puppets using the GEM's 3D 1 http://www.katjaas.nl/personatrix/personatrix.html primitives and Katja created the interpretation and chose the voice effects for the two characters.
The presentation took place on October 2, 2019. During the performance, Jarbas controlled the projected puppets, and Katja interpreted the voices, each character with a different voice effect synthesized in real-time through Personatrix, which was controlled through her modified numpad device called Numberface (Figure 12).
After presenting and explaining the operation of both the hand tracking and voice effect systems, the students were able to ask questions about the mamulengo puppets, about Holland, and about Brazil.
After PdWeekend, all sixteen Maria Oliveira's puppets were purchased by New Blankets and distributed to members of the Pure Data community around the world.
The partnership with Joseph Deken and Katja Vetter was a first step towards the beginning of the internationalization of "A Peleja do Boneco Real com o Boneco Virtual." We had the opportunity to start a conversation for a new version of the cordel in English with the possible participation of Katja Vetter and her two puppets called e (named after the base of the natural logarithm) and i (named after to the unit imaginary number). Hugo Jácome modeled these two Katjas's puppets which were used in the Pure Data/GEM prototype version of the Virtual Puppet system (Figure 13). PdWeekend2019's photo, video, source code published on GitHub 1 , and email records may be used in the future for a new qualitative case study. Furthermore, with other new iterations of development of this puppetry technology, there will be more cases for a collective case study.
As stated by reviewers of this paper, the use of VR/AR/MR technology could be more explored. For example, we can experiment techniques to do the insertion of real Mamulengos into a virtual environment in real-time. This way puppeteers can feel the traditional Mamulengo puppet.
We can also mention as possibilities for future work, the experimentation of other mappings of movements and gestures of the puppeteers' hands in movements and actions of the puppet. We can also try different positioning of the depth sensor or small RGB cameras, to take advantage of the structure of traditional glove puppets. As an example, the sensor can be positioned on the puppets' heads, who would be able to "see" differently and produce sounds automatically, based on what they are viewing. A system for recording the movements of puppets to create animations, was also thought of as an exciting feature by our team artists.
This research also inspired the idea of a VR/AR/MR internet platform, founded on research and actions to connect puppeteers, contribute to the history and the technique of manufacturing and manipulating glove puppets worldwide.
After this experience, artists became interested in keeping working on puppetry and computer science intersections. During informal discussions about the future first public show of the sketch, they appreciated the idea of using a Mamulengo's style tent, and keep it respecting the traditional sizes of the puppets. The hand pose tracking sensor or camera will be visible by the audience and evident in the tent, pointed to the "nude" hand of Virtual Puppet handler (Figure 14). 1 https://github.com/jarbasjacome/MamuLEDs-Pd Our literature review using automatic search was limited to Scopus database, which includes studies from different areas like Leonardo Journal about Arts and Science intersections. For future works, we need to do a new review, including other databases like ACM and IEEE, and add the term "mixed reality" to the search string to enhance the review coverage.

Conclusion
The hand pose tracking system in the sketch analyzed in this study showed limitations for professional use. We highlight the following issues: Leap Motion tracking unreliability, the virtual puppet's lack of tact, the puppeteer's lack of freedom of movement. Still, the experiment had positive reception by some puppetry professionals, the sketch's presentation audience, students, and teachers of the RVA component. In the end, this experience pointed to a promising field of aesthetic experimentation and aroused interest in continuing research by puppetry professionals.
We verified a great conceptual distance between the Virtual Puppet and the traditional mamulengo puppets. Therefore, we accepted the suggestion to use the term discovered by Kennedy Albuquerque: MamuLEDs.
Finally, after the entire journey, we evaluate that this experience contributes effectively to the construction of a tool that develops in direct dialogue with puppetry traditions. In this process, new possibilities arise from the use of VR/AR/MR technologies that intend reality (the world) and representations (languages) through this approximation with this ancestral art. Mamulengo, in turn, may have its visibility redimensioned worldwide through this fruitful encounter with these new realities.