Enhancing the Human with Computers : Ongoing research at the Computer Graphics , Image Processing and Interaction Group

The Computer Graphics, Image Processing and Interaction (CGIP) group at UFRGS concentrates expertise from many different and complementary graphics related domains. In this paper we introduce the group and present our research lines and some ongoing projects. We selected mainly the projects related to 3D interaction and navigation, which includes applications as massive data visualization, surgery planning and simulation, tracking and computer vision algorithms, and modeling approaches for human perception and natural world.


I. HISTORY AND MISSION
The CGIP group started its activities in 1978 and along the years many students pursued their M.Sc.projects on rendering and animation.By the end of the 90's, the group started to grow, aggregating new researchers that brought different expertise.New research topics and application areas as image acquisition and analysis, virtual reality, and visualization of complex data started to be investigated.Many techniques developed by the group became interactive and research efforts started to include different interaction techniques, exploiting the use of non-conventional devices to allow easy manipulation of virtual objects as well as natural navigation in information spaces.
Different factors collaborate for the several academic and practical results obtained by the group: group members have a varied background due to their PhD or pos-doc training in different, highly qualified institutions, and have maintained cooperation projects with well-known groups along the years.Along the years, the group is attracting a number of excellent MSc and PhD candidates.

II. LAB OBJECTIVES
As graphics and image research is a very dynamic environment, the group defines itself today as an interactive visualization research group, where visualization goes beyond the eye to include the touch and the auditory system.Our current main goal is then to use the most out of the computational resources available to enhance human activity with all helpful sensory (not only visual) information in harmony with our living environment while bearing in mind the human tasks the systems are entitled to support.
Such goal touches a number of computational and human aspects, which are covered by the projects in the different research lines pursued by the group members.

III. RESEARCH LINES
The research lines developed within the group encompass several themes, but in the following description we cover only those related to virtual reality and 3D interaction.

A. Non-conventional and 3D interaction
With pervasive computing, the use of keyboard and mouse is no longer the best solution for interaction.The group investigates new possible ways of interfacing people and computers, these ranging from innovative 2D interaction techniques for information visualization to multimodal techniques in 3D immersive environments.It includes interaction with large and high resolution display walls and mobile devices.

B. Interactive visualization and analysis of complex data
Complex data emerge in several domains as images, signals, and domain-related attributes obtained from natural entities or produced by simulation.The group develops techniques for the representation, visualization and analysis of spatial, multidimensional, hierarchical, relational and temporal data.

C. Interaction and animation for games
The game industry needs very efficient algorithms and hardware, as well as innovative interaction without the accuracy required by serious applications.For this reason, they are a great test bed for new algorithms and interaction techniques.The group is exploring new 3D user interfaces for games, and proposing new algorithms for computer animation.

D. Modeling the natural world
CG objects and phenomena can be divided into two major groups: natural or manufactured.Many challenges are involved in the modeling of natural objects, due to their intrinsic complexity and natural beauty.In this research line we tackle the problem of modeling visual patterns in animals and plants, including aging processes and more complex dynamic phenomena adopting a unified approach, where shape and visual attributes are seamless integrated.

E. Image and video processing and analysis
The group has been exploring extraction and summarization of images and videos for content indexing and retrieval, face recognition techniques, and visual data coding and storage.Techniques for real time video and image capturing and processing for video-surgery and augmented reality applications are also being investigated.

A. Minimally invasive surgery simulation
While minimally invasive surgery (MIS) techniques allow minimal postoperative pain, few postoperative adhesions, minimized blood loss, low risk of surgical complications, short hospital stay and early return to normal activities, they are very complex to perform.Physicians must spend several years to acquire the fine motor skills and build mental models of each step of the operations.VR-based training systems can reduce training time and cost as they can be permanently available and offer a great variability of training situations.In this project we deal with physics-based tissue modeling (deformation, cutting, suturing, bleeding) and interaction (3D, stereo, haptics) to develop the next generation of surgical simulators (Fig. 1) [1].

B. VR-and AR-based tools for planning medical interventions
We propose new interactive visualization tools for analyzing 3D medical data and to support diagnosis and treatment planning.Important decision making information, as organ and tumor volumes, or estimation of remaining functional tissue volume after surgery, are obtained from direct interaction on patient specific 3D data (CT or MRI).Part of the visualization is done using mobile display devices, allowing augmented reality views directly on the patient's body [2].Our first case studies are on anatomic hepatectomies (Fig. 2).

C. Interactive visualization of social networks
Social networks have been studied along the years but only recently they have become used as a mean of communication.The group has started to work with social networks as an experimental area for graph visualization techniques.A technique (and tool) named MagnetViz [3] was designed for the interactive manipulation of force-directed graph layouts, allowing users to obtain visualizations based on the graph topology and/or the attributes of its nodes and edges.Users can introduce virtual magnets anywhere in the layout and these can be set to attract nodes and edges that fulfill user-defined criteria.The technique is being improved with social network analysis methods and new 3D interaction techniques.

D. Interactive volume visualization
Many applications deal with volumetric data, which can be visualized either directly, using transfer functions that map data values to color and opacity values, or as surfaces that are extracted from the volume by some thresholding process.Both categories of visualization algorithms have been investigated in successive projects within our group as well as ways of interacting with volumes either carving or adding material as a sculptor [4].Recent results allow interactive visualization of the volume interior displayed as an illustration (Fig. 3) to enhance perception of features.

E. VR-based risk perception analysis in power distribution
Human perception in dangerous environments influences work performance and correlates to accident occurrence.This project, in cooperation with AES-Sul (a large electric power distribution company), aims at evaluating risk perception from behavior in simulated tasks.Complete immersive VR environments in which trainees and professionals will be exposed to dangerous simulated situations while performing maintenance in high-voltage equipments are being built (Fig. 4).We will evaluate possibilities to improve the workers' decision making capability regarding perception of environmental factors during the task.

F. Animation and navigation in virtual environments
The animation of characters in real-time applications is still a challenge when the problem involves attaining a precise location in a virtual world (path-planning) mimicking the human behavior.Likewise, the navigation within virtual environments in a natural and semi-automatic manner is also an open issue.Efficient path-planning methods are being explored to allow the movement of robots or virtual agents in planar environments [5], as well as on complex 3D surfaces with holes and bends (see Fig. 5).

G. Tracking moving objects with adaptation to occlusions
Handling partial and total occlusions of moving object tracking situations is a challenge.In this project, we propose a new adaptive filtering method, named adaptive particle filter (APF), that uses two different operation modes depending on the occlusion of the tracked vehicle.Our approach is designed to detect when a total occlusion starts and ends, as well as to resume object tracking after disocclusions (see Fig. 6).We tested the APF approach in a number of traffic surveillance video sequences, and it tends to be more robust to target occlusions and more accurate than comparable methods in the literature [6].

H. Passive background removal for videoconferences
For a more immersive sensation in videoconferencing applications, a uniform background would be more desirable than a mosaic of different background scenarios.In this context, background removal using a single monocular camera (still the most common setup) may support natural, seamless interaction between remote participants (see Fig. 7).For silhouette extraction, we focused on the upper body (head-shoulders region).Our approach [7] is based on a face detection/tracking algorithm, used to guide a generic Ω-shaped template representing the head-shoulders region.This template defines a search region, and edge-based, color-based and motion-based cues are used to extract the silhouette of the participant within the search region, by maximizing the cost of a path in a graph.

I. Vision-based road inspection
Traffic accidents are a big issue in Brazil, and bad road conditions play an important role.This project focuses on the detection of some visual traffic signs that are crucial for driving safety, such as horizontal lane markings (that delimit the lane boundaries) and vertical traffic signs.In particular, the use of calibrated cameras is important to relate world and camera coordinates, allowing the measurement of distances and also the inclusion of coherent virtual objects for augmented reality experiences.Fig. 8 shows some results on automatic camera calibration based on geometric features of the road, along with a possible AR application for driver assistance systems (insertion of virtual horizontal signs).

J. Human visual aspects: modeling, measurement, simulation
The study of aspects involved in the human visual system have applications in medicine, visualization, human-computer interaction, realistic image synthesis, and so on.This project deals with a wide range of these aspects.We have developed the first practical dynamic model for pupil light reflex, as well as a model for deformation of the iris patterns.Besides the creation of photorealistic animations, these models could be used to screen patients with serious diseases, such as Alzheimer and glaucoma.We have also developed a model for simulation of color vision deficiency (CVD) [8], a condition that affects approximately 200 million people worldwide.This could lead to more inclusive and user-friendly interfaces and visualizations.Complementing this work, we have developed real-time recoloring techniques to allow individuals with CVD to recover, as much as possible, the lost color contrast.Our group has also developed innovative ways of measuring visual acuity and refractive errors (i.e., myopia, hyperopia, and astigmatism) [9], as well as measuring and modeling cataracts in the human eye [10], using programmable displays and interactive techniques.

K. Hysteroscopy video summarization and browsing
Specialists often need to browse through libraries containing many diagnostic hysteroscopy videos searching for similar cases, or even to review the video of one particular case.However, in terms of visual content, diagnostic hysteroscopy videos contain lots of information, but only a reduced number of frames are actually useful for diagnosis/prognosis purposes.In order to facilitate the browsing task, we study techniques for estimating the clinical relevance of video segments in diagnostic hysteroscopies.Basically, we investigate techniques that associate clinical relevance with the attention attracted by a diagnostic hysteroscopy video segment during the video acquisition.In this work, we wish to contribute to the field with summarization and representation methods specific for video hysteroscopies.Preliminary experimental results are promising, and indicate that our methods tend to produce compact video summaries without discarding clinically relevant information [11].

L. Synthesis of patterns in mammals and plant kingdom
This project extends the Clonal Mosaic Model to account for biologically plausible contrasting fur patterns in animals as well as many familiar patterns in the plant kingdom.For patterned animals, we are investigating how to create shape and appearance together, thus generating intelligent textures, since they can adapt to the surface of the object according to geometric information (Fig. 9).For creating patterns in leaves the challenge is assigning consistent texture coordinates since these patterns are often needed in large quantities and with high geometric variation.

M. FitVis: Visualizing heart rate data
The objective of this project is to use data provided by currently available fitness monitoring devices (heart rate, cadence, speed, altitude, longitude, latitude, etc) to create visualizations with high informative value to ordinary users.Through the application of knowledge on time series analytics and information visualization techniques, we attempt to find the relationship among the several available variables and among multiple activities in order to provide better understanding and useful insights on the user's evolution, create projections, support decision-making and in creating new training strategies.To validate our results, we work with data from real activities and the support of fitness and cardiology professionals.

N. High-quality alpha matting, compositing, and filtering
Matting, compositing, and filtering are fundamental image and video processing operations.Image matting aims at extracting foreground elements from images by means of color and opacity (alpha) estimation.It is a fundamental tool for many image and video processing applications, but one that was traditionally slow.We have developed efficient solutions for performing high-quality alpha matting of natural images and videos in real time [12].Our approach is a few orders of magnitude faster than previous ones.Edge-aware filtering is another important image and video processing tool that was notably slow.For it, we have developed an efficient approach that handles, for the first time, full HD video content in realtime [13].

V. FINAL REMARKS
As seen in the previous sections, projects range from image (and video) processing applications to realistic rendering of natural objects, visualization of abstract data, and interaction, covering almost all research topics on graphics domain.This diversity provides an interesting work environment, fostering collaboration between students.Due to these characteristics, the group welcomes MSc and PhD students with varied background but with strong evidence of a research-oriented profile, and also seeks collaboration with other groups to broaden research opportunities.

Fig. 2 .
Fig. 2. Steps in liver surgery planning.The liver is segmented from CT and functional segments are classified from vessel branching directly on the 3D volume.

Fig. 3 .
Fig. 3. Illustration with a focus showing volume interior: center and size are interactively defined.

Fig. 4 .
Fig. 4. Preliminary views of the VR-based simulator for risk perception analysis in electric power distribution (AES-Sul).

Fig. 5 .
Fig.5.Navigation in a complex virtual environment running on a display wall (left); and path-planning on a complex 3D triangle mesh (right).

Fig. 6 .
Fig. 6.Example of a tracked vehicle using the proposed adaptive particle filtering approach.The crosses indicate the initial and final tracked vehicle positions, and the dotted line the tracked vehicle path.The solid line shows the vehicle path interpolated to reduce the occlusion edge effect.

Fig. 7 .
Fig. 7. Example of background substitution using alpha matting: frame (left) and extracted silhouette and replaced background (right).

Fig. 8 .
Fig. 8. Frames illustrating the automatic camera calibration procedure, and the insertion of a virtual horizontal sign.

Fig. 9 .
Fig. 9.In the cow example, geometry and surface appearance are grown together from biologically plausible patterns.Center and right show real and simulated plant patterns.