AI-Driven Responsive GUI for XR: Context-Aware Visual Adaptation

• André Corrêa Santos Insper
• Pedro Bittar Barão Insper
• Rafael Melhado Araujo Lima Insper
• Luciano Pereira Soares Insper

Resumo

Augmented Reality enhances a user’s perception of the real world by overlaying digital information onto the physical environment in real time. One common means of interaction within AR is through Graphical User Interfaces, however, these interfaces can inadvertently occlude important elements of the physical scene. Although window repositioning is straightforward in traditional 2D interfaces, spatial placement in 3D AR environments is more complex, as it must consider factors such as user position, viewing direction, and contextual relevance. This work proposes an approach for the automatic positioning and coloring of GUI elements in AR, enabling interactive panels to dynamically adapt to environmental and contextual conditions. The system adjusts interface placement and appearance in real time based on contextual cues, such as object detection, human presence, and lighting changes. To accomplish this, computer vision techniques are employed to identify and classify objects, while generative AI models interpret high-level contextual information. This enables the interface to reposition itself to avoid occlusions and adjust visual properties, such as color and contrast, for optimal legibility and user comfort. The design and refinement of the system were guided by iterative user testing to ensure usability and effectiveness.

Palavras-chave: Augmented Reality, Graphical User Interfaces, User Interaction, Automatic User Interface Placement, Context-Aware Extended Reality

Referências

Y. Xing, C. Fahy, and J. Shell. Assessing web 2d user interface experiences in mixed reality. Heliyon, 10(11):e31916, 2024. Available [link]

J. G. Grandi, Z. Cao, M. Ogren, and R. Kopper. Design and simulation of next-generation augmented reality user interfaces in virtual reality. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), 23–29, 2021.

T. Hollerer, S. Feiner, D. Hallaway, B. Bell, M. Lanzagorta, D. Brown, S. Julier, Y. Baillot, and L. Rosenblum. User interface management techniques for collaborative mobile augmented reality. Computers & Graphics, 25(5):799–810, 2001. Mixed realities – beyond conventions. Available [link]

C. Santos, B. Miranda, T. Araujo, N. Carneiro, A. Marques, M. Mota, J. Morais, and B. Meiguins. Guidelines for graphical user interface design in mobile augmented reality applications. In Virtual, Augmented and Mixed Reality, S. Lackey and R. Shumaker, Eds. Springer International Publishing, Cham, 71–80, 2016.

A. Ejaz, S. Ali, M. Ejaz, and F. A. Siddiqui. Graphic user interface design principles for designing augmented reality applications. International Journal of Advanced Computer Science and Applications, 10:209–216, 2019.

K. Todi and T. R. Jonker. A framework for computational design and adaptation of extended reality user interfaces. Available [link], 2023. Also at [link]

F. Paterno. User interface design adaptation. In The Encyclopedia of Human-Computer Interaction, 2nd ed. K. Madsen, Ed. Interaction Design Foundation, 2024. Accessed Sep. 17, 2024.

J. Grubert, T. Langlotz, S. Zollmann, and H. Regenbrecht. Towards pervasive augmented reality Context-awareness in augmented reality. IEEE Transactions on Visualization and Computer Graphics, 23(6):1706–1724, 2017.

Z. Stefanidi, G. Margetis, S. Ntoa, and G. Papagiannakis. Real-time adaptation of context-aware intelligent user interfaces for enhanced situational awareness. IEEE Access, 10:23367–23393, 2022.

F. Lu, M. Chen, H. Hsu, P. Deshpande, C. Y. Wang, and B. MacIntyre. Adaptive 3D UI placement in mixed reality using deep reinforcement learning. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24). Association for Computing Machinery, New York, 2024. Available DOI: 10.1145/3613905.3651059

Unity Technologies. Unity. Available [link]

AR Foundation. AR Foundation 6.0.0-pre.5. Available [link]

D. Heaney. Meta will keep looking at giving Quest developers raw camera access. UploadVR, Jul. 2024. Available [link], accessed 2025-06-21.

OpenAI. ChatGPT. Available [link], 2024.

C.-Y. Wang, I.-H. Yeh, and H.-Y. M. Liao. YOLOv9 Learning what you want to learn using programmable gradient information. arXiv preprint arXiv:2402.13616, 2024. Accessed Sept. 18, 2024.

L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth Anything Unleashing the power of large-scale unlabeled data. In CVPR, 2024.

S. Zuffi et al. Human computer interaction Legibility and contrast. IEEE, 2022. Available [link], accessed Nov. 23, 2022.

Apple Inc. Introducing Apple Vision Pro. Available [link], 2023. Accessed 2025-06-21.

Unity Technologies. AsyncGPURReadbackRequest and EncodeToPNG. Unity Discussions, Sep. 2018. Available [link]

S. Liu, P. Lai, D. Tian, and C.-W. Chen. New depth coding techniques with utilization of corresponding video. In Proc. 3DTV Conference, 551–561, 2011.

Black Technology LTD. Living room with people using VR and anchor art. Stable Diffusion, 2024. Available [link]