360RAT: A Tool for Annotating Regions of Interest in 360-degree Videos
Resumo
This paper introduces the software 360RAT as a tool for annotat ing regions of interest (RoIs) in 360-degree videos. These regions represent the portions of the video content that are important for telling a story throughout the video. We believe that this software is an invaluable tool for studying different aspects of 360-degree videos, including what viewers consider relevant and interesting to the experience. As part of this work, we conducted a subjective experiment in which 9 human observers used the proposed soft ware to annotate 11 360-degree videos. As a result, we created a dataset containing a set of annotated 360-degree videos, i.e., videos with marked RoIs and their semantic classification. We present a simple analysis of the annotations gathered with the experiment for a subset of the videos. We noticed that there is a higher agreement of annotations among participants for videos with fewer objects. We also compared the RoI maps with saliency maps computed with the Cube Padding saliency model. We found a strong correlation between RoI maps and computed saliency models, indicating a link between the annotated RoI and the saliency properties of the content.
Palavras-chave:
360-degree video, cinematic VR, attention guiding, user experience
Referências
2018. Computer vision annotation tool. [Online]. https://www.v7labs.com/blog/cvat-guide 5 de maio de 2022
2018. labelImage. [Online]. https://github.com/tzutalin/labelImg 5 de maio de 2022
2018. Labelpix. [Online]. https://github.com/unsignedrant/labelpix. 5 de maio de 2022
2018. Python Video Annotator.. [Online]. https://github.com/unsignedrant/labelpix 5 de maio de 2022
2022. CANNES XR: Showcases. [Online]. https://www.marchedufilm.com/programs/cannes-xr/showcases/. 19 de fevereiro de 2022
2022. Epic Games. [Online]. https://store.epicgames.com/en-US/. 5 de junho de 2022
2022. Planar. [Online]. https://www.planar.com/markets/virtual-production/. 24 de fevereiro de 2022
2022. VR Gorilla. [Online]. https://www.vr-gorilla.com/. 19 de fevereiro de 2022
Lemonia Argyriou, Daphne Economou, and Vassiliki Bouki. 2020. Design methodology for 360 immersive video applications: the case study of a cultural heritage virtual tour. Personal and Ubiquitous Computing (2020), 1–17.
Tewodros Atanaw Biresaw, Tahir Habib Nawaz, James M. Ferryman, and Anthony I. Dell. 2016. ViTBAT: Video tracking and behavior annotation tool. 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) (2016), 295–301.
J. Bremmers. 2017. Narrative cues within cinematic virtual reality: An exploratory study of narrative cues within the content and motives of virtual reality developers. Master’s thesis. http://hdl.handle.net/2105/42782
J Brillhart. 2016. VR & Cinema.
Cullen Brown, Ghanshyam Bhutra, Mohamed Suhail, Qinghong Xu, and Eric D Ragan. 2017. Coordinating attention and cooperation in multi-user virtual reality narratives. In 2017 IEEE Virtual Reality (VR). IEEE, 377–378.
Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand. 2018. What do different evaluation metrics tell us about saliency models? IEEE transactions on pattern analysis and machine intelligence 41, 3 (2018), 740–757.
Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, and Min Sun. 2018. Cube Padding for Weakly-Supervised Saliency Prediction in 360° Videos. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (Jun 2018). https://doi.org/10.1109/cvpr.2018.00154
Cisco. 2018. Cisco Annual Report 2018-2023. [link] 12 de fevereiro de 2021
Savino Dambra, Giuseppe Samela, Lucile Sassatelli, Romaric Pighetti, Ramon Aparicio-Pardo, and Anne-Marie Pinna-Déry. 2018. Film editing: New levers to improve VR streaming. In Proceedings of the 9th ACM Multimedia Systems Conference. 27–39.
Raphael S. de Abreu, Joel André Ferreira dos Santos, and Débora Christina Muchaluat-Saade. 2021. Sensory Effect Extraction for 360° Media Content. Proceedings of the Brazilian Symposium on Multimedia and the Web (2021).
Abhishek Dutta and Andrew Zisserman. 2019. The VGG Image Annotator (VIA). ArXiv abs/1904.10699 (2019).
Mylène Farias, Myllena Prado, and Lucas Althoff. 2021. 360RAT - 360 - ROI Annotator Tool. https://gitlab.com/gpds-unb/360rat
Hannes Fassold and Barnabás Takács. 2019. Towards Automatic Cinematography and Annotation for 360° Video. Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video (2019).
Florian Groh, Dominik Schörkhuber, and Margrit Gelautz. 2020. A tool for semi-automatic ground truth annotation of traffic videos. electronic imaging 2020 (2020), 200–1–200–7.
Hou-Ning Hu, Yen-Chen Lin, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, and Min Sun. 2017. Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Huawei. 2017. VR/AR Huawei white paper. [link] 12 de fevereiro de 2021
Sebastian Knorr, Cagri Ozcinar, Colm O Fearghail, and Aljosa Smolic. 2018. Director’s cut: a combined dataset for visual attention analysis in cinematic VR content. In Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production. 1–10.
Matthias Kummerer, Thomas S. A. Wallis, and Matthias Bethge. 2018. Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics. In Proceedings of the European Conference on Computer Vision (ECCV).
Liang-Han Lin, Hao-Kai Wen, Man-Hsin Kao, Evelyn Chen, Tse-Han Lin, and Ming Ouhyoung. 2020. Label360: An Annotation Interface for Labeling Instance-Aware Semantic Labels on Panoramic Full Images. SIGGRAPH Asia 2020 Posters (2020).
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. Microsoft COCO: Common Objects in Context. arXiv:1405.0312 [cs.CV]
C Milk. 2016. Chris Milk: The birth of virtual reality as an art form [Video File].
OpenCV. 2015. Open Source Computer Vision Library. https://opencv.org/.
Jayesh S Pillai, Azif Ismail, and Herold P Charles. 2017. Grammar of VR story-telling: visual cues. In Proceedings of the virtual reality international conference-laval virtual 2017. 1–4.
python. 2021. PyQt5 5.15.4. https://pypi.org/project/PyQt5/.
Michel Reilhac. 2016. Presence Design and Spatial Writing in Virtual Reality.
Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. 2007. LabelMe: A Database and Web-Based Tool for Image Annotation. International Journal of Computer Vision 77 (2007), 157–173.
Lucile Sassatelli, Anne-Marie Pinna-Déry, Marco Winckler, Savino Dambra, Giuseppe Samela, Romaric Pighetti, and Ramon Aparicio-Pardo. 2018. Snapchanges: a dynamic editing strategy for directing viewer’s attention in streaming virtual reality videos. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces. 1–5.
Ana Serrano, Vincent Sitzmann, Jaime Ruiz-Borau, Gordon Wetzstein, Diego Gutierrez, and Belen Masia. 2017. Movie editing and cognitive event segmentation in virtual reality video. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–12.
Anting Shen. 2016. BeaverDam : Video Annotation Tool for Computer Vision Training Labels.
Afshin Taghavi, Aliehsan Samiei, Anahita Mahzari, Ryan McMahan, Ravi Prakash, Mylene Farias, and Marcelo Carvalho. 2019. A taxonomy and dataset for 360° videos. 273–278. https://doi.org/10.1145/3304109.3325812
Carl Vondrick, Donald J. Patterson, and Deva Ramanan. 2012. Efficiently Scaling up Crowdsourced Video Annotation. International Journal of Computer Vision 101 (2012), 184–204.
Shang-Ta Yang, Chi-Han Peng, Peter Wonka, and Hung kuo Chu. 2018. PanoAnnotator: a semi-automatic tool for indoor panorama layout annotation. SIGGRAPH Asia 2018 Posters (2018).
2018. labelImage. [Online]. https://github.com/tzutalin/labelImg 5 de maio de 2022
2018. Labelpix. [Online]. https://github.com/unsignedrant/labelpix. 5 de maio de 2022
2018. Python Video Annotator.. [Online]. https://github.com/unsignedrant/labelpix 5 de maio de 2022
2022. CANNES XR: Showcases. [Online]. https://www.marchedufilm.com/programs/cannes-xr/showcases/. 19 de fevereiro de 2022
2022. Epic Games. [Online]. https://store.epicgames.com/en-US/. 5 de junho de 2022
2022. Planar. [Online]. https://www.planar.com/markets/virtual-production/. 24 de fevereiro de 2022
2022. VR Gorilla. [Online]. https://www.vr-gorilla.com/. 19 de fevereiro de 2022
Lemonia Argyriou, Daphne Economou, and Vassiliki Bouki. 2020. Design methodology for 360 immersive video applications: the case study of a cultural heritage virtual tour. Personal and Ubiquitous Computing (2020), 1–17.
Tewodros Atanaw Biresaw, Tahir Habib Nawaz, James M. Ferryman, and Anthony I. Dell. 2016. ViTBAT: Video tracking and behavior annotation tool. 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) (2016), 295–301.
J. Bremmers. 2017. Narrative cues within cinematic virtual reality: An exploratory study of narrative cues within the content and motives of virtual reality developers. Master’s thesis. http://hdl.handle.net/2105/42782
J Brillhart. 2016. VR & Cinema.
Cullen Brown, Ghanshyam Bhutra, Mohamed Suhail, Qinghong Xu, and Eric D Ragan. 2017. Coordinating attention and cooperation in multi-user virtual reality narratives. In 2017 IEEE Virtual Reality (VR). IEEE, 377–378.
Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand. 2018. What do different evaluation metrics tell us about saliency models? IEEE transactions on pattern analysis and machine intelligence 41, 3 (2018), 740–757.
Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, and Min Sun. 2018. Cube Padding for Weakly-Supervised Saliency Prediction in 360° Videos. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (Jun 2018). https://doi.org/10.1109/cvpr.2018.00154
Cisco. 2018. Cisco Annual Report 2018-2023. [link] 12 de fevereiro de 2021
Savino Dambra, Giuseppe Samela, Lucile Sassatelli, Romaric Pighetti, Ramon Aparicio-Pardo, and Anne-Marie Pinna-Déry. 2018. Film editing: New levers to improve VR streaming. In Proceedings of the 9th ACM Multimedia Systems Conference. 27–39.
Raphael S. de Abreu, Joel André Ferreira dos Santos, and Débora Christina Muchaluat-Saade. 2021. Sensory Effect Extraction for 360° Media Content. Proceedings of the Brazilian Symposium on Multimedia and the Web (2021).
Abhishek Dutta and Andrew Zisserman. 2019. The VGG Image Annotator (VIA). ArXiv abs/1904.10699 (2019).
Mylène Farias, Myllena Prado, and Lucas Althoff. 2021. 360RAT - 360 - ROI Annotator Tool. https://gitlab.com/gpds-unb/360rat
Hannes Fassold and Barnabás Takács. 2019. Towards Automatic Cinematography and Annotation for 360° Video. Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video (2019).
Florian Groh, Dominik Schörkhuber, and Margrit Gelautz. 2020. A tool for semi-automatic ground truth annotation of traffic videos. electronic imaging 2020 (2020), 200–1–200–7.
Hou-Ning Hu, Yen-Chen Lin, Ming-Yu Liu, Hsien-Tzu Cheng, Yung-Ju Chang, and Min Sun. 2017. Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Huawei. 2017. VR/AR Huawei white paper. [link] 12 de fevereiro de 2021
Sebastian Knorr, Cagri Ozcinar, Colm O Fearghail, and Aljosa Smolic. 2018. Director’s cut: a combined dataset for visual attention analysis in cinematic VR content. In Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production. 1–10.
Matthias Kummerer, Thomas S. A. Wallis, and Matthias Bethge. 2018. Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics. In Proceedings of the European Conference on Computer Vision (ECCV).
Liang-Han Lin, Hao-Kai Wen, Man-Hsin Kao, Evelyn Chen, Tse-Han Lin, and Ming Ouhyoung. 2020. Label360: An Annotation Interface for Labeling Instance-Aware Semantic Labels on Panoramic Full Images. SIGGRAPH Asia 2020 Posters (2020).
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. Microsoft COCO: Common Objects in Context. arXiv:1405.0312 [cs.CV]
C Milk. 2016. Chris Milk: The birth of virtual reality as an art form [Video File].
OpenCV. 2015. Open Source Computer Vision Library. https://opencv.org/.
Jayesh S Pillai, Azif Ismail, and Herold P Charles. 2017. Grammar of VR story-telling: visual cues. In Proceedings of the virtual reality international conference-laval virtual 2017. 1–4.
python. 2021. PyQt5 5.15.4. https://pypi.org/project/PyQt5/.
Michel Reilhac. 2016. Presence Design and Spatial Writing in Virtual Reality.
Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. 2007. LabelMe: A Database and Web-Based Tool for Image Annotation. International Journal of Computer Vision 77 (2007), 157–173.
Lucile Sassatelli, Anne-Marie Pinna-Déry, Marco Winckler, Savino Dambra, Giuseppe Samela, Romaric Pighetti, and Ramon Aparicio-Pardo. 2018. Snapchanges: a dynamic editing strategy for directing viewer’s attention in streaming virtual reality videos. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces. 1–5.
Ana Serrano, Vincent Sitzmann, Jaime Ruiz-Borau, Gordon Wetzstein, Diego Gutierrez, and Belen Masia. 2017. Movie editing and cognitive event segmentation in virtual reality video. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–12.
Anting Shen. 2016. BeaverDam : Video Annotation Tool for Computer Vision Training Labels.
Afshin Taghavi, Aliehsan Samiei, Anahita Mahzari, Ryan McMahan, Ravi Prakash, Mylene Farias, and Marcelo Carvalho. 2019. A taxonomy and dataset for 360° videos. 273–278. https://doi.org/10.1145/3304109.3325812
Carl Vondrick, Donald J. Patterson, and Deva Ramanan. 2012. Efficiently Scaling up Crowdsourced Video Annotation. International Journal of Computer Vision 101 (2012), 184–204.
Shang-Ta Yang, Chi-Han Peng, Peter Wonka, and Hung kuo Chu. 2018. PanoAnnotator: a semi-automatic tool for indoor panorama layout annotation. SIGGRAPH Asia 2018 Posters (2018).
Publicado
07/11/2022
Como Citar
PRADO, Myllena; ALTHOFF, Lucas; ALAMGEER, Sana; RODRIGUES E SILVA, Alessandro; PRAKASH, Ravi; CARVALHO, Marcelo M.; FARIAS, Mylène C. Q..
360RAT: A Tool for Annotating Regions of Interest in 360-degree Videos. In: BRAZILIAN SYMPOSIUM ON MULTIMEDIA AND THE WEB (WEBMEDIA), 28. , 2022, Curitiba.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 291-299.
