xGroupware: Supporting Collaborative Cross-Reality Environments using Multiagents System

this paper presents xGroupware, a cross-reality meetingware that coordinates some activities to support remote workgroup. xGroupware was developed based on a multiagent system architecture that integrates different technologies for enabling message exchange between virtual and real worlds users using wearable computers. Meetingware, Collaboration, Cross-Reality, Wearable Computers, Multiagents system.


INTRODUCTION
Meetings are essential for structuring and coordinating workgroup [1].Many different types of technology can be used to support this type of interaction, whether in face to face or at a distance.Collaborative 3D environments stand out from other meetingware tools for teams working at a distance by promoting a strong sense of presence and engagement among participants [2].
3D collaborative environments facilitate interactions between group participants, promoting the exchange of information and ideas, and group problem solving, thus motivating participation and a sense of shared responsibility [3], [4], [5].These environments enable communication via text, voice, gestures and facial expressions.Communication is not restricted to the 3D environment itself, but rather extends to other virtual means, for example, e-mail.However, there is a communication disruption from the virtual to the real [4].
Cross-reality environments, resulting from the integration of collaborative 3D environments with ubiquitous computing environments [7], enable communication between the real and virtual worlds.Within these environments, information is collected by a network of real and virtual sensors, propagated to both worlds, interpreted and presented to users.
xGroupware creates a cross-reality environment which aims to support meeting planning and execution in the context of distance workgroup where the collaborative 3D environment is the shared one.The development of xGroupware was based on an architecture that integrates different technologies to extend collaboration in cross-reality environments.This Multiagent Systems (MAS) architecture enables the description of a system from the perspective of its constituent units [8].
A MAS is modeled as a collection of hardware or software entities, called agents, that enjoy the following properties: (a) reactivityagents perceive their environments and respond to changes that occur in it; (b) autonomy -agents operate without the direct intervention of human or others and have control over their actions and internal state; (c) social abilityagents interact with other agents using some kind of agent communication language; (d) pro-activenessagents are able to exhibit goal-directed behavior and take the initiative [9].In xGroupware, each system element is an agent with well-defined goals and responsibilities.Each agent evaluates its situation and acts, without external control, based on its set of behaviors.Even counting on individual agents, whose behaviors are relatively simple, these systems have complex behaviors due to interactions between its agents.
This architecture was proposed for facilitating the integration of the following technologies:  Collaborative 3D environments;  Sensors and actuators used in a ubiquitous computing environment [10];  Wearable computers -clothing and accessories with computer devices connected to sensors and actuators; due to their proximity to the body, these wearables are part of the personal space of the users [11];  Tangible interfaces, employing physical artifacts to represent and control digital information [12]; and  MASs, which allow the simulation of intelligent behavior, such as problem solving and learning ability, using approaches such as neural networks and Markov Models.In the next section, related work is discussed.Section III describes xGroupware application.The proposed architecture and prototype implementation are detailed in Section IV.Section V introduces two xGroupware scenarios.Conclusions and future work are presented in Section VI.

II. RELATED WORK
Due to the persistence and high level of immersion offered by virtual worlds, several tools were created for socialization, community formation, and meeting execution and training.An example of this type of tool is the Sametime 3D [13] application developed by IBM in the OpenSim 3D virtual environment [14].It is used for remote meetings, training, brainstorming sessions (Fig. 1) and planning activities.Avatars communicate using audio, text or both, and share presentations and notes using a virtual flipchart, and vote to make group decisions [15].Figure 1.The framework for virtual notes is used in Sametime 3D brainstorming sessions [15].
inSpace [16] and TwinSpace [17] are projects from Georgia Institute of Technology that has developed tools that integrate real-world users with those in the virtual world.TwinSpace is an infrastructure for cross-reality environments that combines interactive spaces and virtual worlds where employees participate in meetings remotely in a ubiquitous computing environment.
The inSpace project includes elements that support crossreality group collaboration.For example, a table and a display that enables multiple users in real or virtual environments are used to share information and services.Fig. 2 shows some of the features used to control presentations in inSpace.
The Augmented Collaboration in Mixed Environments (ACME) system is a videoconferencing application that employs a collaborative 3D environment, augmented reality and gesture recognition techniques [18].In this system, participants in the 3D environment or the real world are viewed through different devices so that they can interact and share virtual objects.Fig. 3 shows an example of meeting with ACME, where participants share a virtual object.The work presented in this paper differentiates from the aforementioned work for using wearable computers in crossreality environment interaction, systematizing some workgroup coordination activities, and by adopting an MASbased architecture.

III. XGROUPWARE
xGroupware is a cross-reality application that enables communication between the real and virtual worlds through message exchange between agents.Using sensors and actuators, participants identify who is present in the virtual and real worlds, invite other participants, confirm their presence at a meeting, and receive confirmation and cancellation notices regarding a scheduled meeting.xGroupware implemented a cross-reality architecture for developing collaborative cross-reality environments entitled xAgent [19].
xGroupware only support meetings that take place in the virtual world."Scheduled meetings" are previously planned meetings that are coordinated by the Meeting Coordinator Agent.On the other hand, any virtual world participant may call a real world participant to join in an "unscheduled meeting".
For the real world, three bracelets were prototyped.The Arduino family was used as the technology for prototyping these bracelets.Lilypad Arduino [20] is a microcontroller board designed for wearables and e-textiles.It is sewn to fabric and similarly mounted power supplies, sensors and actuators with conductive thread.Not using mobile phone technology was a project decision given our choice for prototyping using Arduino's open-source electronics platform [21].
These wearables interface with the participants through sensors and actuators (Fig. 4).LEDs and a vibeboard were used for showing data originating from the virtual world and the Meeting Manager agent.Whenever a LED is on, it means that a specific participant is logged into the virtual world and she is in the meetingware area.Whenever LEDs blink slower, there is a call for an unscheduled meeting.If they are blinking faster, there is a call for a scheduled one.Whenever the LEDs are blinking and the vibeboard is active, a meeting is about to take place.Participants reply to these events by pushing the Lilypad button.A single click on the button means that the participant will attend the meeting and a double click means that she won't.A network of XBee modules [20] makes it possible to receive and send the events that occur in real world.All these components are powered by a lithium battery and are sewn by conductive thread.Within the virtual world, a meetingware was prototyped in Second Life [23].xGroupware connects the virtual meetingware to the real world meetingware.The following virtual world objects were inserted into the meetingware for supporting the cross-reality communication.
 Head-up display (HUD): Used in the virtual world to individually interact with a participant in the browser, in a similar way as the bracelet used in the real world.The bottom right of Fig. 5 shows the HUD as a small square filled with participants' names.A white circle in the left column of the HUD indicates that the participants are in real world and a light blue circle indicates that she was summoned for an unscheduled meeting.A green circle on the right column shows that the participant replied that she will attend to the meeting and a red circle means that she replied that she won't attend.Finally, white circle shows that the participant was called for a meeting and she hasn't answered it yet.
 Meetings Display: Display in the virtual room which presents information about the next meeting and the status of the last meeting.Fig. 5 shows the next scheduled meeting information on the left display of the Meeting Display and a meeting's cancelled status on the right one.
 Presence sensor: an invisible virtual object that detects the avatars those are online and located in the meetingware area.xGroupware enables agents' synchronous communication for meeting support and to systematize tasks previously performed by participants.In the following section, issues involved in implementing the system are detailed.

IV. ARCHITECTURE AND IMPLEMENTATION
Considering the cross-reality context described above, agents' properties -autonomy, communication, reactivity and pro-activity -coincide with the desired characteristics of the real and virtual objects.Therefore, a MAS-based architecture was implemented.Fig. 6 presents the xGroupware architecture in terms of its agents.The crossreality interaction is promoted by the Messenger and Communicators who propagate events between both worlds.The Messenger receives messages from different sources and redistributes them to the correspondent agents, according to event types.Communicators are also agents, which act as bridges between the Messenger and the agents contained within real-world and virtual objects.Participants use wearables for interacting from their real world offices.3D objects are modeled in the virtual world and are injected with scripted behaviors for enabling interaction with participants and other 3D objects.2. Enables the participant to invite a real-world participant to join an unscheduled meeting.
3. Enables the participant to confirm its participation in a scheduled meeting.

4.
Informs the participant about the presence status of participants in a meeting.

Virtual Meeting Room Communicator
1. Coordinates message exchanges between the Messenger and virtual world objects.

Agent Responsibilities
Presence Sensor 1. Informs other agents about the presence of virtual world group members.
Meeting Display 1. Presents information to the participant about the next scheduled meeting.
Bracelet 1. Informs a real-world communicator of a participant's presence in the real environment.
2. Informs the participant about the presence of virtual world participants.
3. Enables the participant to confirm the existence of a scheduled meeting.
4. Enables the participant to confirm the existence of an unscheduled meeting.
Each agent acts according to its own responsibilities, reacting to propagated events that are associated to its scripted behaviors.In addition to responding to incoming messages, an agent also propagates events in the environment or solicits information from any participant or agent involved in the meetings.
xGroupware makes use of the proposed architecture to implement agents in the real world and the virtual world.To this end, it integrates different technologies, as detailed in Table II and Fig. 7.

Agent Technology
Messenger Implemented using Spring framework using Java language.Uses sockets for interaction with the Ultra Communicator, a web server to communicate with the 3D Virtual Meeting Room Communicator and the Communicator for the exchange of FIPA (Foundation for Intelligent Physical Agents) [24] messages for the FIPA Communicator agent.
Ultra Communicator Java language.
Meeting Manager Implemented using Jade framework [25], specifically for MASs using Jade4Spring.FIPA Communicator

Virtual Meeting Room Communicator
Objects in the virtual world are implemented using the Second Life platform, in the Linden Scripting (programming) Language (LSL).

Bracelet
Wearables (bracelets) based on Arduino technology and XBee modules for wireless communications.

The communication between the Messenger and xGroupware objects via the Communicators employs:
 sockets for communication with wearables,  a web server to exchange messages with the virtual world, Second Life, and  a communicator to interact with the Meeting Manager agent.The Ultra Communicator (real world communicator wearable,) implemented in Java, uses a serial port that receives and sends messages to the bracelets through an XBee antenna.This communicator uses a Server Socket service for sending and receiving messages to the Messenger.The bracelets were implemented using Arduino technology.
The virtual world communicator is an object implemented in the LSL programming language.Given that in LSL it is only possible to exchange messages with other systems by using an HTTP protocol, the web server was implemented within the Messenger.
The Meeting Manager is an agent that uses the Jade platform.Accordingly, the communicator that operates between the Messenger and the Meeting Manager was also implemented in Jade, using the FIPA standard for MAS communications [24].The proposed architecture also enables adding features and functionalities by coupling external systems.The format for external systems integration is defined in terms of their technological characteristics.
As a result of its adaptability to different contexts involving interactions between real and virtual worlds, generality is one of the properties of the xGroupware system.Another property to be highlighted is its flexibility, which emerges from the possibility of including different real objects and external systems in the composition of a specific solution.

V. XGROUPWARE SCENARIOS
The following scenarios exemplify the collaboration that takes place within the xGroupware environment.In the first case virtual world meetings are scheduled while in the second one, they take place at the invitation of another participant.For scheduled meetings, the Meeting Manager is responsible for organizing meetings, including their confirmation, as illustrated in the following scenario: Denise, Victor and Ana have a meeting scheduled in the 3D Virtual Meeting Room at 2:00 p.m.At 1:30 p.m., Denise, who is working in this room, receives a request to confirm her participation at the 2:00 p.m. meeting, to which she responds positively.At 1:40 p.m., Ana arrives at her office and turns on her bracelet.After a few minutes, she also receives, via the bracelet, a request to confirm her participation at the 2:00 p.m. meeting.Ana replies she cannot attend the meeting.Subsequently, all participants are informed that the meeting had been canceled.
The Meeting Manager is responsible for requesting attendance confirmations from meeting participants and, based on each response, confirms or cancels the event.In case of cancellation, the agent reschedules the meeting and sends the new date and time to all participants.
Requests for participation confirmations start thirty minutes before the beginning of the scheduled meeting.The Meeting Manager agent sends the RSVP requests to the real and virtual world.Participants confirm their participation by either pressing the HUD or the bracelet's button, as can be seen in Fig. 4. As it was described in the scenario above, at 1:30 p.m., the Meeting Manager agent sends a confirmation request to Denise and Ana.Ana receives it when she turned on her bracelet.The bracelet signals for a meeting attendance RSVP by means of light signals.She then answers whether she will attend the meeting by using the selfsame bracelet.Similarly, a HUD is used in the virtual world to display attendance RSVP requests for meetings in which specific participants need to confirm their presence.Ana presses a button on her bracelet to reply that she will not be able to attend to the meeting.In the virtual room, the Meeting Display informs about the next meeting status (Fig. 5), indicating its confirmation or cancellation.Therefore, in the scenario described above, Denise and Ana received the meeting cancellation notice by HUD and the bracelet, respectively.
The second scenario illustrates the taking place of an unscheduled meeting, in which Denise invites her colleague, Ana, to join her in the virtual room: Ana and Denise participate in a working group that uses the 3D Virtual Meeting Room.Denise is in the virtual room and, by observing its HUD, notices that Ana is in her office (that is equipped with a presence sensor).Denise then takes the opportunity to invite Ana to join her in the virtual world to conclude a pending job.Blinking LEDS on Ana's bracelet inform her about Denise's invitation.She replies positively, pressing the button on her bracelet, indicating that it she will meet her in the virtual room.
Only participants in the virtual world are able to invite participants in the real world to attend meetings.Whenever an unscheduled meeting is requested, the invited one has 5 minutes for confirming her participation before the request is cancelled.Ana confirms her participation and the meeting takes place when Ana joins Denise in the Virtual Meeting Room.
In these two scenarios, agents communicate between themselves, and react and take action based on the events originating on the two worlds.Agent Modeling Language (AML) [26] sequence diagrams are used for representing the interactions between these agents.Figure 9 shows the messages exchanged between the agents on the unscheduled meeting scenario.When Denise requests an unscheduled call, her (agent) HUD perceives this event and the (agent) Messenger propagates it by using the (agent) Communicators.Thus, the Meeting Manager can arrange the meeting, and the bracelet in real world informs Ana of this request.In the same way, when Ana confirms her participation, the Meeting Manager confirms the meeting and Denise is notified by the HUD.As it was decided to use multiagent systems and different platforms for interfacing both worlds, the Communicators and the Messenger exchange the same message between them (Figure 9).This message is called "event" and it distinguishes one from other because the use of the arguments in terms of a n-tuple.A n-tuple in this message arguments determines in this order: the agent which initializes the event, the origin (Meeting Manager, real or virtual world), the event and other parameters.For example, the message 9 in Figure 9 means that the participant 02 in the real world (Ana) confirmed that she will attend the meeting call from participant 01 (Denise).

VI. CONCLUSION AND FURTHER WORK
This paper presents a cross-reality meetingware entitled xGroupware.It presents other meetingware for analyzing related work and other implementation strategies.The xGroupware architecture presented explains the agents' responsibilities, the interaction between the real and virtual objects and the technology involved for implementing this cross-reality meetingware.
xGroupware aims to support the planning and execution of virtual meetings.Using a 3D virtual environment as a shared space for collaboration, xGroupware brings to the real-world features that are also available in the virtual environment, enabling participants in both worlds to interact.
The related work presented in this paper show some meetingware in virtual worlds, real world and cross-reality environments.xGroupware distinguishes from these meetingware according to the use of wearable computers in the real world and its interaction with the virtual world by agents.
The proposed architecture aims to implement cross-reality solutions that involve the exchange of messages between objects in the real and virtual worlds.The choice for using MAS as the basis for this architecture is due to the capacity of the agents to perceive and interact, in a pro-active and autonomous manner, with other agents and the environment in which they are inserted.Different implementation technologies and communications protocols are adopted for facilitating the exchange of messages.
Scheduled and unscheduled meeting scenarios presented illustrates some of the systems functionalities and elucidate agents' behaviors and the communication between them.AML sequence diagrams are used for representing the communication in these scenarios.
For further enhancements of the proposed architecture, other application contexts will be evaluated.The usability of the bracelets and other, wearables and other tangible devices is one of the issues to be investigated.To do so, wearable variations will be introduced (e.g.shirts and gloves instead of bracelets) to compare attributes, such as interactivity and effectiveness.

Figure 8 .
Figure 8. Meeting attendance confirmation using the bracelet in the real world and by selecting a button on a HUD in the virtual world.

Figure 9 .
Figure 9. AML Sequence Diagram of an unscheduled meeting call.
Table I specifies the responsibilities of xGroupware agents.

TABLE I .
AGENT RESPONSIBILITIES