AirSnip: Gesture-Based AR Screenshot Enhanced by GenAI - Oliveira, Robson; Simões, Francisco (Anais Estendidos da Conference on Graphics, Patterns and Images (SIBGRAPI); 2025: Anais Estendidos da XXXVIII Conference on Graphics, Patterns and Images, 2025)
This paper presents the development of an interactive tool for image capture in Augmented Reality (AR) environments, based on a custom mid-air hand gesture that mimics the behavior of a snipping tool. The system allows users to delimit a rectangular area in space using a gesture, capture a real-world view through the Meta Quest 3’s passthrough camera, and generate a 3D virtual frame with the resulting image. These frames can be manipulated using natural gestures (e.g., push, pinch) and are enhanced with personalized descriptions generated by a Large Language Model (LLM) via Google Gemini, with audio output through Text-to-Speech (TTS). Built using Unity and the Meta XR SDK, the prototype offers a seamless and intuitive interface for storytelling, education, and spatial memory. Although user studies are planned for future work, the tool demonstrates the technical feasibility of integrating gesture recognition, spatial rendering, and Generative AI in a unified AR experience.