A Taxonomy of Methods, Tools, and Approaches for Enabling Collaborative Annotation
Resumo
Collaborative annotation reduces the time and cost of the annotation task and increases data quality. For knowledge-intensive contexts, collaborative annotation is an interesting approach to combine expert users' knowledge with domain data. This work was motivated by a use case where expert users' tacit knowledge associated with externalized data can produce new domain knowledge. That new knowledge can inform AI-based systems and make them more accurate for that specific domain. To identify and understand the collaborative annotation process' dimensions, especially when the annotation is used to support AI models, we defined the following research question: "What are the methods, tools, and solutions to allow collaborative annotation?". We did a systematic literature review, using five specific questions to help us answer the main research question. Our findings allowed us to identify different domains that adopt this approach, tools for collaborative annotation, players present in the process, and strategies for collaborative annotation. We summarize the findings in a taxonomy to represent aspects related to the characterization of annotation, the collaboration itself, and user experience strategies to support this process. We conclude the paper suggesting open opportunities for future research and implications for the design of an AI-based system to support experts in knowledge-intensive processes.
Palavras-chave:
collaborative annotation, human-in-the-loop, artificial intelligence
Referências
Giuseppe Abrami, Manuel Stoeckel, and Alexander Mehler. 2020. TextAnnotator: A UIMA based tool for the simultaneous and collaborative annotation of texts. In Proceedings of the 12th Language Resources and Evaluation Conference. 891–900.
Samreen Anjum, Chi Lin, and Danna Gurari. 2021. CrowdMOT: Crowdsourcing strategies for tracking multiple objects in videos. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1–25.
Enrique Manjavacas Arévalo and Peter Petré. 2017. Enabling annotation of historical corpora in an asynchronous collaborative environment. In Proceedings of the 2nd International Conference on Digital Access to Textual Cultural Heritage. 9–14.
Ala Atrash, Marie-Hélène Abel, and Claude Moulin. 2014. Supporting organizational learning with collaborative annotation. In International Conference on Knowledge Management and Information Sharing. 237–244.
Danilo Avola, Paolo Bottoni, Marco Laureti, Stefano Levialdi, and Emanuele Panizzi. 2010. Managing groups and group annotations in madcow. In Databases in Networked Information Systems: 6th International Workshop, DNIS 2010, Aizu-Wakamatsu, Japan, March 29-31, 2010. Proceedings 6. Springer, 194–209.
Sanjana Babu, V Gowtham, L Parmila Mary Sophia, and P Pabitha. 2013. A collaborative tagging system with formal concept analysis. In 2013 International Conference on Recent Trends in Information Technology (ICRTIT). IEEE, 1–6.
Federico Bartoli, Giuseppe Lisanti, Lorenzo Seidenari, and Alberto Del Bimbo. 2017. PACE: prediction-based annotation for crowded environments. In Proceedings of the 2017 ACM on international conference on multimedia retrieval. 121–124.
Valerio Basile, Bos Johan, Evang Kilian, Venhuizen Noortje, 2012. A platform for collaborative semantic annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Association for Computational Linguistics, 92–96.
Tobias Baur, Alexander Heimerl, Florian Lingenfelser, Johannes Wagner, Michel F Valstar, Björn Schuller, and Elisabeth André. 2020. eXplainable cooperative machine learning with NOVA. KI-Künstliche Intelligenz 34 (2020), 143–164.
Siegfried Benkner, Chris Borckholder, Marian Bubak, Yuriy Kaniovskyi, Richard Knight, Martin Koehler, Spiros Koulouzis, Piotr Nowakowski, and Steven Wood. 2013. A Cloud-based framework for collaborative data management in the VPH-Share Project. In 2013 27th International Conference on Advanced Information Networking and Applications Workshops. IEEE, 1203–1210.
Jesper Bergman and Oliver B Popov. 2022. The Digital Detective’s Discourse-A toolset for forensically sound collaborative dark web content annotation and collection. Journal of Digital Forensics, Security and Law 17, 1 (2022), 5.
Sreyasee Das Bhattacharjee, Bala Venkatram Balantrapu, William Tolone, and Ashit Talukder. 2017. Identifying extremism in social media with multi-view context-aware subset optimization. In 2017 IEEE international conference on big data (big data). IEEE, 3638–3647.
Sreyasee Das Bhattacharjee, Ashit Talukder, Ehab Al-Shaer, and Pratik Doshi. 2017. Prioritized active learning for malicious url detection using weighted text-based features. In 2017 IEEE International Conference on Intelligence and Security Informatics (ISI). IEEE, 107–112.
Sreyasee Das Bhattacharjee, Ashit Talukder, and Bala Venkatram Balantrapu. 2017. Active learning based news veracity detection with feature weighting and deep-shallow fusion. In 2017 IEEE International Conference on Big Data (Big Data). IEEE, 556–565.
Enea Bionda, Francesca Soldan, Mattia Cabiati, and Luciano Martini. 2022. The Smart Grids Innovation Accelerator–SGIA: an international open platform to boost smart grids innovation through knowledge sharing. In 2022 AEIT International Annual Conference (AEIT). IEEE, 1–6.
Kalina Bontcheva, Hamish Cunningham, Ian Roberts, Angus Roberts, Valentin Tablan, Niraj Aswani, and Genevieve Gorrell. 2013. GATE Teamware: a web-based, collaborative text annotation framework. Language Resources and Evaluation 47 (2013), 1007–1029.
Mariam Bouchakwa, Yassine Ayadi, and Ikram Amous. 2020. A review on visual content-based and users’ tags-based image annotation: methods and techniques. Multimedia Tools and Applications 79 (2020), 21679–21741.
Federico Cabitza, Carla Simone, and Marco P Locatelli. 2012. Supporting artifact-mediated discourses through a recursive annotation tool. In Proceedings of the 17th ACM international conference on Supporting group work. 253–262.
M Cai, WY Zhang, and K Zhang. 2010. ManuHub: a semantic web system for ontology-based service management in distributed manufacturing environments. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 41, 3 (2010), 574–582.
Matheus de L Calache and Cléver RG de Farias. 2020. Graphical and collaborative annotation support for semantic Web services. In 2020 IEEE International Conference on Software Architecture Companion (ICSA-C). IEEE, 210–217.
Yu-Chien Chen, Ren-Hung Hwang, and Cheng-Yu Wang. 2012. Development and evaluation of a Web 2.0 annotation system as a learning tool in an e-learning environment. Computers & Education 58, 4 (2012), 1094–1105.
Isabella Chiari, Aldo Gangemi, Elisabetta Jezek, Alessandro Oltramari, Guido Vetere, and Laure Vieu. 2013. An open knowledge base for italian language in a collaborative perspective. In Proceedings of the 1st International Workshop on Collaborative Annotations in Shared Environment: metadata, vocabularies and techniques in the Digital Humanities. 1–6.
Stefano Cresci, Davide Gazzè, Angelica Lo Duca, Andrea Marchetti, and Maurizio Tesconi. 2015. Geo data annotator: a web framework for collaborative annotation of geographical datasets. In Proceedings of the 24th International Conference on World Wide Web. 23–24.
Valeria Croce, Gabriella Caroti, Livio De Luca, A Piemonte, and P Véron. 2020. Semantic annotations on heritage models: 2D/3D approaches and future research challenges. In International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. 43. Copernicus GmbH, 829–836.
Stamatia Dasiopoulou, Eirini Giannakidou, Georgios Litos, Polyxeni Malasioti, and Yiannis Kompatsiaris. 2011. A survey of semantic image and video annotation tools. Knowledge-Driven Multimedia Information Extraction and Ontology Evolution: Bridging the Semantic Gap (2011), 196–239.
Tobias Daudert, Manel Zarrouk, and Brian Davis. 2019. CoSACT: A collaborative tool for fine-grained sentiment annotation and consolidation of text. In Proceedings of the first workshop on financial technology and natural language processing. 34–39.
Nicoletta Dessì, Giuliano Ferrentino, Emanuele Pascariello, and Barbara Pes. 2015. Towards Ontology-Enabled BioContexts for Bioinformatics Research. In 2015 IEEE 24th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises. IEEE, 236–238.
Nicoletta Dessì, Gabriele Milia, Emanuele Pascariello, and Barbara Pes. 2016. COWB: A cloud-based framework supporting collaborative knowledge management within biomedical communities. Future Generation Computer Systems 54 (2016), 399–408.
Claudio Di Ciccio, Andrea Marrella, and Alessandro Russo. 2015. Knowledge-Intensive Processes: Characteristics, Requirements and Analysis of Contemporary Approaches. Journal on Data Semantics 4, 1 (March 2015), 29–57. DOI: 10.1007/s13740-014-0038-4
Brian Dorn, Larissa B Schroeder, and Adam Stankiewicz. 2015. Piloting TrACE: Exploring spatiotemporal anchored collaboration in asynchronous learning. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. 393–403.
Reim Doumat, Elöd Egyed-Zsigmond, and Jean-Marie Pinon. 2010. User trace-based recommendation system for a digital archive. In Case-Based Reasoning. Research and Development: 18th International Conference on Case-Based Reasoning, ICCBR 2010, Alessandria, Italy, July 19-22, 2010. Proceedings 18. Springer, 360–374.
Reim Doumat, Elöd Egyed-Zsigmond, and Jean-Marie Pinon. 2013. A web 2.0 archive to access, annotate and retrieve manuscripts. Multimedia tools and applications 64 (2013), 97–117.
Yuqi Fang, Delong Zhu, Niyun Zhou, Li Liu, and Jianhua Yao. 2021. PiPo-Net: A Semi-automatic and Polygon-based Annotation Method for Pathological Images. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2978–2984.
Karën Fort. 2016. Collaborative annotation for reliable natural language processing: Technical and sociological aspects. John Wiley & Sons.
Joaquín Gayoso-Cabada, Amelia Sanz-Cabrerizo, and José-Luis Sierra. 2013. @ note: an electronic tool for academic readings. In Proceedings of the 1st International Workshop on Collaborative Annotations in Shared Environment: metadata, vocabularies and techniques in the Digital Humanities. 1–4.
Jose Nuno Gomes-Pereira, Vincent Auger, Kolja Beisiegel, Robert Benjamin, Melanie Bergmann, David Bowden, Pal Buhl-Mortensen, Fabio C De Leo, Gisela Dionísio, Jennifer M Durden, 2016. Current and future trends in marine image annotation software. Progress in Oceanography 149 (2016), 106–120.
Anna Goy, Diego Magro, Giovanna Petrone, Claudia Picardi, and Marino Segnan. 2016. Ontology-driven collaborative annotation in shared workspaces. Future Generation Computer Systems 54 (2016), 435–449.
Jonatas S. Grosman, Pedro H.T. Furtado, Ariane M.B. Rodrigues, Guilherme G. Schardong, Simone D.J. Barbosa, and Hélio C.V. Lopes. 2020. Eras: Improving the quality control in the annotation process for Natural Language Processing tasks. Information Systems 93 (2020), 101553. DOI: 10.1016/j.is.2020.101553
Peter Haase, Tobias Mathäß, Michael Schmidt, Andreas Eberhart, and Ulrich Walther. 2010. Semantic technologies for enterprise cloud management. In The Semantic Web–ISWC 2010: 9th International Semantic Web Conference, ISWC 2010, Shanghai, China, November 7-11, 2010, Revised Selected Papers, Part II 9. Springer, 98–113.
Allan Hanbury. 2008. A survey of methods for image annotation. Journal of Visual Languages & Computing 19, 5 (2008), 617–627. DOI: 10.1016/j.jvlc.2008.01.002
Robert J Hendley, Russell Beale, Chris P Bowers, Christos Georgousopoulos, Charalampos Vassiliou, Petridis Sergios, Ralf Moeller, Eric Karstens, and Dimitris Spiliotopoulos. 2014. CASAM: collaborative human-machine annotation of multimedia. Multimedia tools and applications 70 (2014), 1277–1308.
Winfried Höhn, Hans-Günter Schmidt, and Hendrik Schöneberg. 2013. Semiautomatic recognition and georeferencing of places in early maps. In Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries. 335–338.
Shun’ya Iwasawa, Hiroki Hanaoka, Takuya Matsuzaki, Yusuke Miyao, and Jun’ichi Tsujii. 2011. A collaborative annotation between human annotators and a statistical parser. In 5th Linguistic Annotation Workshop. 56–64.
Rui Jesus, Luís Bastião Silva, and Carlos Costa. 2022. Quantitative Measurements Extraction and Annotation Optimization in Whole Slide Imaging. In 2022 IEEE International Symposium on Medical Measurements and Applications (MeMeA). IEEE, 1–6.
Arshad Khan, Thanassis Tiropanis, and David Martin. 2017. Crowd-annotation and LoD-based semantic indexing of content in multi-disciplinary web repositories to improve search results. In Proceedings of the Australasian Computer Science Week Multiconference. 1–12.
Shah Khusro, Mumtaz Khan, and Irfan Ullah. 2016. Collaborative video annotation based on ontological themes, temporal duration and pointing regions. In Proceedings of the 10th International Conference on Informatics and Systems. 121–126.
Barbara Kitchenham, O Pearl Brereton, David Budgen, Mark Turner, John Bailey, and Stephen Linkman. 2009. Systematic literature reviews in software engineering–a systematic literature review. Information and software technology 51, 1 (2009), 7–15.
Gary Klein, Ben Shneiderman, Robert R. Hoffman, and Kenneth M. Ford. 2017. Why Expertise Matters: A Response to the Challenges. IEEE Intelligent Systems 32, 6 (Nov. 2017), 67–73. DOI: 10.1109/MIS.2017.4531230
István Koren, Petru Nicolaescu, and Ralf Klamma. 2015. Collaborative drawing annotations on web videos. In Engineering the Web in the Big Data Era: 15th International Conference, ICWE 2015, Rotterdam, The Netherlands, June 23-26, 2015, Proceedings 15. Springer, 671–674.
Nesrine Ksentini, Mohamed Zarka, Anis Ben Ammar, and Adel M Alimi. 2012. Toward an assisted context based collaborative annotation. In 2012 10th International Workshop on Content-Based Multimedia Indexing (CBMI). IEEE, 1–6.
Amel Ksibi, Nizar Elleuch, Anis Ben Ammar, and Adel M Alimi. 2011. Semi-automatic soft collaborative annotation for semantic video indexing. In 2011 IEEE EUROCON-International Conference on Computer as a Tool. IEEE, 1–6.
Kostiantyn Kucher, Carita Paradis, Magnus Sahlgren, and Andreas Kerren. 2017. Active learning and visual analytics for stance classification with ALVA. ACM Transactions on Interactive Intelligent Systems (TiiS) 7, 3 (2017), 1–31.
E Li, Shuaijun Wang, Chengyang Li, Dachuan Li, Xiangbin Wu, and Qi Hao. 2020. Sustech points: A portable 3d point cloud interactive annotation platform system. In 2020 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1108–1115.
Dongyu Liu, Sarah Alnegheimish, Alexandra Zytek, and Kalyan Veeramachaneni. 2022. MTV: Visual analytics for detecting, investigating, and annotating anomalies in multivariate time series. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1–30.
André Lourenço, Hugo Plácido Da Silva, Carlos Carreiras, Ana Priscila Alves, and Ana LN Fred. 2014. A web-based platform for biosignal visualization and annotation. Multimedia Tools and Applications 70 (2014), 433–460.
Mathias Lux and Michael Riegler. 2013. Annotation of endoscopic videos on mobile devices: a bottom-up approach. In Proceedings of the 4th ACM Multimedia Systems Conference. 141–145.
Lena Maier-Hein, Tobias Ross, Janek Gröhl, Ben Glocker, Sebastian Bodenstedt, Christian Stock, Eric Heim, Michael Götz, S Wirkert, Hannes Kenngott, 2016. Crowd-algorithm collaboration for large-scale endoscopic image annotation with confidence. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19. Springer, 616–623.
Raphaël Marée, Benjamin Stévens, Loïc Rollus, Natacha Rocks, Xavier Moles Lopez, Isabelle Salmon, Didier Cataldo, and Louis Wehenkel. 2013. A rich internet application for remote visualization and collaborative annotation of digital slides in histology and cytology. In Diagnostic Pathology, Vol. 8. BioMed Central, 1–4.
Abir Masmoudi, Emna Mezghani, Hatem Bellaaj, Khalil Drira, and Mohamed Jmaiel. 2017. A web-based knowledge management system for scientific research team. In 2017 IEEE 26th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE). IEEE, 296–301.
Lucas McCullum, Hasan Saeed, Benjamin Moody, Diane Perry, Eric Gottlieb, Tom Pollard, Xavier Borrat Frigola, Qiao Li, Gari Clifford, Roger G Mark, . An Open-Source Platform for Collaborative Annotation of Physiological Waveforms. ( ), .
Marcio Ferreira Moreno, Wallas Henrique Sousa Dos Santos, Rodrigo Costa Mesquita Santos, and Renato Fontoura De Gusmao Cerqueira. 2018. Supporting knowledge creation through has: The hyperknowledge annotation system. In 2018 IEEE International Symposium on Multimedia (ISM). IEEE, 239–246.
Julie A Murphy. 2021. Collaborative Annotation: Tools for Enhancing Learning and Scholarly Communication. Serials Review 47, 3-4 (2021), 157–162.
Steven R Ness, Alex Lerch, and George Tzanetakis. 2011. Strategies for orca call retrieval to support collaborative annotation of a large archive. In 2011 IEEE 13th International Workshop on Multimedia Signal Processing. IEEE, 1–6.
Mariana Neves and Jurica Ševa. 2021. An extensive review of tools for manual annotation of documents. Briefings in bioinformatics 22, 1 (2021), 146–163.
Lucas ES Oliveira, Caroline P Gebeluca, Adalniza MP Silva, Claudia MC Moro, Sadid A Hasan, and Oladimeji Farri. 2017. A statistics and UMLS-based tool for assisted semantic annotation of Brazilian clinical documents. In 2017 IEEE International conference on bioinformatics and biomedicine (BIBM). IEEE, 1072–1078.
Yuxin Pan, Jianwu Fang, Jian Dou, Zhen Ye, and Jianru Xue. 2018. Research on Human-Machine Collaborative Annotation for Traffic Scene Data. In 2018 Chinese Automation Congress (CAC). IEEE, 2900–2905.
Frederic Papazian, R. Robert Bossy, and Claire Nédellec. 2012. AlvisAE: a collaborative Web text annotation editor for knowledge acquisition. In LAW VI’12 - Sixth Linguistic Annotation Workshop.
Vishwajeet Pattanaik, Shweta Suran, and Dirk Draheim. 2019. Enabling social information exchange via dynamically robust annotations. In Proceedings of the 21st International Conference on Information Integration and Web-based Applications & Services. 176–184.
Martín Pérez-Pérez, Daniel Glez-Peña, Florentino Fdez-Riverola, and Anália Lourenço. 2015. Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects. Computer methods and programs in biomedicine 118, 2 (2015), 242–251.
Johann Poignant, Mateusz Budnik, Hervé Bredin, Claude Barras, Mickael Stefas, Pierrick Bruneau, Gilles Adda, Laurent Besacier, Hazim Ekenel, Gil Francopoulo, 2016. The CAMOMILE collaborative annotation platform for multi-modal, multi-lingual and multi-media documents. In LREC 2016 Conference.
Elena Potapenko, Pascal Kovacs, and Elisabeth Burr. 2013. “From Leipzig into the Romania”-Environment for collaborative annotation and knowledge creation. In Proceedings of the 1st International Workshop on Collaborative Annotations in Shared Environment: metadata, vocabularies and techniques in the Digital Humanities. 1–4.
Daniel Racoceanu and Frédérique Capron. 2015. Towards semantic-driven high-content image analysis: An operational instantiation for mitosis detection in digital histopathology. Computerized Medical Imaging and Graphics 42 (2015), 2–15.
Lozana Rossenova, Zoe Schubert, Richard Vock, Lucia Sohmen, Lukas Günther, Paul Duchesne, and Ina Blümel. 2022. Collaborative annotation and semantic enrichment of 3D media: a FOSS toolchain. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries. 1–5.
Christopher Rydell and Joakim Lindblad. 2021. CytoBrowser: a browser-based collaborative annotation platform for whole slide images. F1000Research 10 (03 2021), 226. DOI: 10.12688/f1000research.51916.1
Aziret Satybaldiev, Peter Hevesi, Marco Hirsch, Vitor Fortes Rey, and Paul Lukowicz. 2019. CoAT: a web-based, collaborative annotation tool. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. 814–818.
Marinette Savonnet, Eric Leclercq, and Pierre Naubourg. 2015. eclims: An extensible and dynamic integration framework for biomedical information systems. IEEE journal of biomedical and health informatics 20, 6 (2015), 1640–1649.
Roger Schaer, Sebastian Otálora, Oscar Jimenez-del Toro, Manfredo Atzori, and Henning Müller. 2019. Deep learning-based retrieval system for gigapixel histopathology cases and the open access literature. Journal of pathology informatics 10, 1 (2019), 19.
Kleberson A Serique, Alan Snyder, Debra Willrett, Daniel L Rubin, and Dilvan A Moreira. 2012. Using the semantic web and web apps to connect radiologists and oncologists. In 2012 IEEE 21st International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. IEEE, 480–485.
Hal Shin, Natanael Exe, Ujwal Dutta, Tanka Raj Joshi, Sagar Verma, and Siddharth Gupta. 2022. Europa: Increasing Accessibility of Geospatial Datasets. In IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium. 6848–6851. DOI: 10.1109/IGARSS46834.2022.9884440
Ehsan Jahangirzadeh Soure, Emily Kuang, Mingming Fan, and Jian Zhao. 2021. CoUX: collaborative visual analysis of think-aloud usability test videos for digital interfaces. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 643–653.
Addison YS Su, Stephen JH Yang, Wu-Yuin Hwang, and Jia Zhang. 2010. A Web 2.0-based collaborative annotation system for enhancing knowledge sharing in collaborative learning environments. Computers & Education 55, 2 (2010), 752–766.
Dan Su and Pascale Fung. 2012. Personalized music emotion classification via active learning. In Proceedings of the second international ACM workshop on Music information retrieval with user-centered and multimodal strategies. 57–62.
Jaana Takis, AQM Saiful Islam, Christoph Lange, and Sören Auer. 2015. Crowdsourced semantic annotation of scientific publications and tabular data in PDF. In Proceedings of the 11th International Conference on Semantic Systems. 1–8.
Nikolaos Tsipas, Panagiotis Zapartas, Lazaros Vrysis, and Charalampos Dimoulas. 2015. Augmenting social multimedia semantic interaction through audio-enhanced web-tv services. In Proceedings of the Audio Mostly 2015 on Interaction With Sound. 1–7.
Jasper RR Uijlings, Mykhaylo Andriluka, and Vittorio Ferrari. 2020. Panoptic Image Annotation with a Collaborative Assistant. In Proceedings of the 28th ACM International Conference on Multimedia. 3302–3310.
Victoria Uren, Philipp Cimiano, José Iria, Siegfried Handschuh, Maria Vargas-Vera, Enrico Motta, and Fabio Ciravegna. 2006. Semantic annotation for knowledge management: Requirements and a survey of the state of the art. Journal of Web Semantics 4, 1 (2006), 14–28.
Byron C Wallace, Kevin Small, Carla E Brodley, Joseph Lau, and Thomas A Trikalinos. 2012. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT international health informatics symposium. 819–824.
Haiyan Wang, Yanjie Zhou, Bing Zhou, and Zongmin Wang. 2021. A Novel Method for Detection of ECG with Deep Learning. In 2021 7th International Conference on Computer and Communications (ICCC). IEEE, 631–635.
Jessica Z Wang, Amy X Zhang, and David R Karger. 2022. Designing for Engaging with News using Moral Framing towards Bridging Ideological Divides. Proceedings of the ACM on Human-Computer Interaction 6, GROUP (2022), 1–23.
Nicholas Warren, Benjamin Garrard, Elliot Staudt, and Jelena Tesic. 2018. Transfer learning of deep neural networks for visual collaborative maritime asset identification. In 2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC). IEEE, 246–255.
Antoine Widlöcher and Yann Mathet. 2012. The glozz platform: A corpus annotation and mining tool. In Proceedings of the 2012 ACM symposium on Document engineering. 171–180.
Stefan Wilk, Stephan Kopf, and Wolfgang Effelsberg. 2016. Collaborative annotation of videos relying on weak consistency. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 12, 3 (2016), 1–23.
Jan Witowski, Jongmun Choi, Soomin Jeon, Doyun Kim, Joowon Chung, John Conklin, Maria Gabriela Figueiro Longo, Marc D Succi, and Synho Do. 2021. MarkIt: a collaborative artificial intelligence annotation platform leveraging blockchain for medical imaging research. Blockchain in Healthcare Today (2021).
Joanna Wolfe. 2002. Annotation technologies: A software and research review. Computers and Composition 19, 4 (2002), 471–497.
Jialun Wu, Anyu Mao, Xinrui Bao, Haichuan Zhang, Zeyu Gao, Chunbao Wang, Tieliang Gong, and Chen Li. 2021. Pimip: An open source platform for pathology information management and integration. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2088–2095.
Xuehong Wu, Junwen Duan, Jianhua Li, and Min Li. 2021. nPTAS: A Novel Platform for Text Annotation and Service. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 544–549.
Zhicheng Yang, Zhenjie Cao, Yanbo Zhang, Peng Chang, Shibin Wu, Lingyun Huang, Wei Xu, Mei Han, Jing Xiao, Mingxiang Wu, 2021. MABEL: An AI-powered mammographic breast lesion diagnostic system. In 2020 IEEE International Conference on E-health Networking, Application & Services (HEALTHCOM). IEEE, 1–7.
Chih-Hao Yu, Tudor Groza, and Jane Hunter. 2013. Reasoning on crowd-sourced semantic annotations to facilitate cataloguing of 3D artefacts in the cultural heritage domain. In The Semantic Web–ISWC 2013: 12th International Semantic Web Conference, Sydney, NSW, Australia, October 21-25, 2013, Proceedings, Part II 12. Springer, 228–243.
Jiaxin Yu, Florian Wellmann, Simon Virgo, Marven von Domarus, Mingze Jiang, Joyce Schmatz, and Bastian Leibe. 2023. Superpixel segmentations for thin sections: evaluation of methods to enable the generation of machine learning training data sets. Computers & Geosciences 170 (2023), 105232.
Beste F Yuksel, Pooyan Fazli, Umang Mathur, Vaishali Bisht, Soo Jung Kim, Joshua Junhee Lee, Seung Jung Jin, Yue-Ting Siu, Joshua A Miele, and Ilmi Yoon. 2020. Human-in-the-loop machine learning to increase video accessibility for visually impaired and blind users. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 47–60.
Lishi Zhang, Chenghan Fu, and Jia Li. 2018. Collaborative annotation of semantic objects in images with multi-granularity supervisions. In Proceedings of the 26th ACM international conference on Multimedia. 474–482.
Christos Zigkolis, Symeon Papadopoulos, George Filippou, Yiannis Kompatsiaris, and Athena Vakali. 2014. Collaborative event annotation in tagged photo collections. Multimedia Tools and Applications 70 (2014), 89–118.
Tobias Zimmermann, Markus Weber, Marcus Liwicki, and Didier Stricker. 2012. CoVidA: pen-based collaborative video annotation. In Proceedings of the 1st International Workshop on Visual Interfaces for Ground Truth Collection in Computer Vision Applications. 1–6.
Damien E Zomahoun and Kokou Yetongnon. 2014. EMERGSEM: emergent semantic and recommendation system for image retrieval. In 2014 Tenth International Conference on Signal-Image Technology and Internet-Based Systems. IEEE, 256–263.
Samreen Anjum, Chi Lin, and Danna Gurari. 2021. CrowdMOT: Crowdsourcing strategies for tracking multiple objects in videos. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (2021), 1–25.
Enrique Manjavacas Arévalo and Peter Petré. 2017. Enabling annotation of historical corpora in an asynchronous collaborative environment. In Proceedings of the 2nd International Conference on Digital Access to Textual Cultural Heritage. 9–14.
Ala Atrash, Marie-Hélène Abel, and Claude Moulin. 2014. Supporting organizational learning with collaborative annotation. In International Conference on Knowledge Management and Information Sharing. 237–244.
Danilo Avola, Paolo Bottoni, Marco Laureti, Stefano Levialdi, and Emanuele Panizzi. 2010. Managing groups and group annotations in madcow. In Databases in Networked Information Systems: 6th International Workshop, DNIS 2010, Aizu-Wakamatsu, Japan, March 29-31, 2010. Proceedings 6. Springer, 194–209.
Sanjana Babu, V Gowtham, L Parmila Mary Sophia, and P Pabitha. 2013. A collaborative tagging system with formal concept analysis. In 2013 International Conference on Recent Trends in Information Technology (ICRTIT). IEEE, 1–6.
Federico Bartoli, Giuseppe Lisanti, Lorenzo Seidenari, and Alberto Del Bimbo. 2017. PACE: prediction-based annotation for crowded environments. In Proceedings of the 2017 ACM on international conference on multimedia retrieval. 121–124.
Valerio Basile, Bos Johan, Evang Kilian, Venhuizen Noortje, 2012. A platform for collaborative semantic annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Association for Computational Linguistics, 92–96.
Tobias Baur, Alexander Heimerl, Florian Lingenfelser, Johannes Wagner, Michel F Valstar, Björn Schuller, and Elisabeth André. 2020. eXplainable cooperative machine learning with NOVA. KI-Künstliche Intelligenz 34 (2020), 143–164.
Siegfried Benkner, Chris Borckholder, Marian Bubak, Yuriy Kaniovskyi, Richard Knight, Martin Koehler, Spiros Koulouzis, Piotr Nowakowski, and Steven Wood. 2013. A Cloud-based framework for collaborative data management in the VPH-Share Project. In 2013 27th International Conference on Advanced Information Networking and Applications Workshops. IEEE, 1203–1210.
Jesper Bergman and Oliver B Popov. 2022. The Digital Detective’s Discourse-A toolset for forensically sound collaborative dark web content annotation and collection. Journal of Digital Forensics, Security and Law 17, 1 (2022), 5.
Sreyasee Das Bhattacharjee, Bala Venkatram Balantrapu, William Tolone, and Ashit Talukder. 2017. Identifying extremism in social media with multi-view context-aware subset optimization. In 2017 IEEE international conference on big data (big data). IEEE, 3638–3647.
Sreyasee Das Bhattacharjee, Ashit Talukder, Ehab Al-Shaer, and Pratik Doshi. 2017. Prioritized active learning for malicious url detection using weighted text-based features. In 2017 IEEE International Conference on Intelligence and Security Informatics (ISI). IEEE, 107–112.
Sreyasee Das Bhattacharjee, Ashit Talukder, and Bala Venkatram Balantrapu. 2017. Active learning based news veracity detection with feature weighting and deep-shallow fusion. In 2017 IEEE International Conference on Big Data (Big Data). IEEE, 556–565.
Enea Bionda, Francesca Soldan, Mattia Cabiati, and Luciano Martini. 2022. The Smart Grids Innovation Accelerator–SGIA: an international open platform to boost smart grids innovation through knowledge sharing. In 2022 AEIT International Annual Conference (AEIT). IEEE, 1–6.
Kalina Bontcheva, Hamish Cunningham, Ian Roberts, Angus Roberts, Valentin Tablan, Niraj Aswani, and Genevieve Gorrell. 2013. GATE Teamware: a web-based, collaborative text annotation framework. Language Resources and Evaluation 47 (2013), 1007–1029.
Mariam Bouchakwa, Yassine Ayadi, and Ikram Amous. 2020. A review on visual content-based and users’ tags-based image annotation: methods and techniques. Multimedia Tools and Applications 79 (2020), 21679–21741.
Federico Cabitza, Carla Simone, and Marco P Locatelli. 2012. Supporting artifact-mediated discourses through a recursive annotation tool. In Proceedings of the 17th ACM international conference on Supporting group work. 253–262.
M Cai, WY Zhang, and K Zhang. 2010. ManuHub: a semantic web system for ontology-based service management in distributed manufacturing environments. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 41, 3 (2010), 574–582.
Matheus de L Calache and Cléver RG de Farias. 2020. Graphical and collaborative annotation support for semantic Web services. In 2020 IEEE International Conference on Software Architecture Companion (ICSA-C). IEEE, 210–217.
Yu-Chien Chen, Ren-Hung Hwang, and Cheng-Yu Wang. 2012. Development and evaluation of a Web 2.0 annotation system as a learning tool in an e-learning environment. Computers & Education 58, 4 (2012), 1094–1105.
Isabella Chiari, Aldo Gangemi, Elisabetta Jezek, Alessandro Oltramari, Guido Vetere, and Laure Vieu. 2013. An open knowledge base for italian language in a collaborative perspective. In Proceedings of the 1st International Workshop on Collaborative Annotations in Shared Environment: metadata, vocabularies and techniques in the Digital Humanities. 1–6.
Stefano Cresci, Davide Gazzè, Angelica Lo Duca, Andrea Marchetti, and Maurizio Tesconi. 2015. Geo data annotator: a web framework for collaborative annotation of geographical datasets. In Proceedings of the 24th International Conference on World Wide Web. 23–24.
Valeria Croce, Gabriella Caroti, Livio De Luca, A Piemonte, and P Véron. 2020. Semantic annotations on heritage models: 2D/3D approaches and future research challenges. In International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. 43. Copernicus GmbH, 829–836.
Stamatia Dasiopoulou, Eirini Giannakidou, Georgios Litos, Polyxeni Malasioti, and Yiannis Kompatsiaris. 2011. A survey of semantic image and video annotation tools. Knowledge-Driven Multimedia Information Extraction and Ontology Evolution: Bridging the Semantic Gap (2011), 196–239.
Tobias Daudert, Manel Zarrouk, and Brian Davis. 2019. CoSACT: A collaborative tool for fine-grained sentiment annotation and consolidation of text. In Proceedings of the first workshop on financial technology and natural language processing. 34–39.
Nicoletta Dessì, Giuliano Ferrentino, Emanuele Pascariello, and Barbara Pes. 2015. Towards Ontology-Enabled BioContexts for Bioinformatics Research. In 2015 IEEE 24th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises. IEEE, 236–238.
Nicoletta Dessì, Gabriele Milia, Emanuele Pascariello, and Barbara Pes. 2016. COWB: A cloud-based framework supporting collaborative knowledge management within biomedical communities. Future Generation Computer Systems 54 (2016), 399–408.
Claudio Di Ciccio, Andrea Marrella, and Alessandro Russo. 2015. Knowledge-Intensive Processes: Characteristics, Requirements and Analysis of Contemporary Approaches. Journal on Data Semantics 4, 1 (March 2015), 29–57. DOI: 10.1007/s13740-014-0038-4
Brian Dorn, Larissa B Schroeder, and Adam Stankiewicz. 2015. Piloting TrACE: Exploring spatiotemporal anchored collaboration in asynchronous learning. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. 393–403.
Reim Doumat, Elöd Egyed-Zsigmond, and Jean-Marie Pinon. 2010. User trace-based recommendation system for a digital archive. In Case-Based Reasoning. Research and Development: 18th International Conference on Case-Based Reasoning, ICCBR 2010, Alessandria, Italy, July 19-22, 2010. Proceedings 18. Springer, 360–374.
Reim Doumat, Elöd Egyed-Zsigmond, and Jean-Marie Pinon. 2013. A web 2.0 archive to access, annotate and retrieve manuscripts. Multimedia tools and applications 64 (2013), 97–117.
Yuqi Fang, Delong Zhu, Niyun Zhou, Li Liu, and Jianhua Yao. 2021. PiPo-Net: A Semi-automatic and Polygon-based Annotation Method for Pathological Images. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2978–2984.
Karën Fort. 2016. Collaborative annotation for reliable natural language processing: Technical and sociological aspects. John Wiley & Sons.
Joaquín Gayoso-Cabada, Amelia Sanz-Cabrerizo, and José-Luis Sierra. 2013. @ note: an electronic tool for academic readings. In Proceedings of the 1st International Workshop on Collaborative Annotations in Shared Environment: metadata, vocabularies and techniques in the Digital Humanities. 1–4.
Jose Nuno Gomes-Pereira, Vincent Auger, Kolja Beisiegel, Robert Benjamin, Melanie Bergmann, David Bowden, Pal Buhl-Mortensen, Fabio C De Leo, Gisela Dionísio, Jennifer M Durden, 2016. Current and future trends in marine image annotation software. Progress in Oceanography 149 (2016), 106–120.
Anna Goy, Diego Magro, Giovanna Petrone, Claudia Picardi, and Marino Segnan. 2016. Ontology-driven collaborative annotation in shared workspaces. Future Generation Computer Systems 54 (2016), 435–449.
Jonatas S. Grosman, Pedro H.T. Furtado, Ariane M.B. Rodrigues, Guilherme G. Schardong, Simone D.J. Barbosa, and Hélio C.V. Lopes. 2020. Eras: Improving the quality control in the annotation process for Natural Language Processing tasks. Information Systems 93 (2020), 101553. DOI: 10.1016/j.is.2020.101553
Peter Haase, Tobias Mathäß, Michael Schmidt, Andreas Eberhart, and Ulrich Walther. 2010. Semantic technologies for enterprise cloud management. In The Semantic Web–ISWC 2010: 9th International Semantic Web Conference, ISWC 2010, Shanghai, China, November 7-11, 2010, Revised Selected Papers, Part II 9. Springer, 98–113.
Allan Hanbury. 2008. A survey of methods for image annotation. Journal of Visual Languages & Computing 19, 5 (2008), 617–627. DOI: 10.1016/j.jvlc.2008.01.002
Robert J Hendley, Russell Beale, Chris P Bowers, Christos Georgousopoulos, Charalampos Vassiliou, Petridis Sergios, Ralf Moeller, Eric Karstens, and Dimitris Spiliotopoulos. 2014. CASAM: collaborative human-machine annotation of multimedia. Multimedia tools and applications 70 (2014), 1277–1308.
Winfried Höhn, Hans-Günter Schmidt, and Hendrik Schöneberg. 2013. Semiautomatic recognition and georeferencing of places in early maps. In Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries. 335–338.
Shun’ya Iwasawa, Hiroki Hanaoka, Takuya Matsuzaki, Yusuke Miyao, and Jun’ichi Tsujii. 2011. A collaborative annotation between human annotators and a statistical parser. In 5th Linguistic Annotation Workshop. 56–64.
Rui Jesus, Luís Bastião Silva, and Carlos Costa. 2022. Quantitative Measurements Extraction and Annotation Optimization in Whole Slide Imaging. In 2022 IEEE International Symposium on Medical Measurements and Applications (MeMeA). IEEE, 1–6.
Arshad Khan, Thanassis Tiropanis, and David Martin. 2017. Crowd-annotation and LoD-based semantic indexing of content in multi-disciplinary web repositories to improve search results. In Proceedings of the Australasian Computer Science Week Multiconference. 1–12.
Shah Khusro, Mumtaz Khan, and Irfan Ullah. 2016. Collaborative video annotation based on ontological themes, temporal duration and pointing regions. In Proceedings of the 10th International Conference on Informatics and Systems. 121–126.
Barbara Kitchenham, O Pearl Brereton, David Budgen, Mark Turner, John Bailey, and Stephen Linkman. 2009. Systematic literature reviews in software engineering–a systematic literature review. Information and software technology 51, 1 (2009), 7–15.
Gary Klein, Ben Shneiderman, Robert R. Hoffman, and Kenneth M. Ford. 2017. Why Expertise Matters: A Response to the Challenges. IEEE Intelligent Systems 32, 6 (Nov. 2017), 67–73. DOI: 10.1109/MIS.2017.4531230
István Koren, Petru Nicolaescu, and Ralf Klamma. 2015. Collaborative drawing annotations on web videos. In Engineering the Web in the Big Data Era: 15th International Conference, ICWE 2015, Rotterdam, The Netherlands, June 23-26, 2015, Proceedings 15. Springer, 671–674.
Nesrine Ksentini, Mohamed Zarka, Anis Ben Ammar, and Adel M Alimi. 2012. Toward an assisted context based collaborative annotation. In 2012 10th International Workshop on Content-Based Multimedia Indexing (CBMI). IEEE, 1–6.
Amel Ksibi, Nizar Elleuch, Anis Ben Ammar, and Adel M Alimi. 2011. Semi-automatic soft collaborative annotation for semantic video indexing. In 2011 IEEE EUROCON-International Conference on Computer as a Tool. IEEE, 1–6.
Kostiantyn Kucher, Carita Paradis, Magnus Sahlgren, and Andreas Kerren. 2017. Active learning and visual analytics for stance classification with ALVA. ACM Transactions on Interactive Intelligent Systems (TiiS) 7, 3 (2017), 1–31.
E Li, Shuaijun Wang, Chengyang Li, Dachuan Li, Xiangbin Wu, and Qi Hao. 2020. Sustech points: A portable 3d point cloud interactive annotation platform system. In 2020 IEEE Intelligent Vehicles Symposium (IV). IEEE, 1108–1115.
Dongyu Liu, Sarah Alnegheimish, Alexandra Zytek, and Kalyan Veeramachaneni. 2022. MTV: Visual analytics for detecting, investigating, and annotating anomalies in multivariate time series. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1–30.
André Lourenço, Hugo Plácido Da Silva, Carlos Carreiras, Ana Priscila Alves, and Ana LN Fred. 2014. A web-based platform for biosignal visualization and annotation. Multimedia Tools and Applications 70 (2014), 433–460.
Mathias Lux and Michael Riegler. 2013. Annotation of endoscopic videos on mobile devices: a bottom-up approach. In Proceedings of the 4th ACM Multimedia Systems Conference. 141–145.
Lena Maier-Hein, Tobias Ross, Janek Gröhl, Ben Glocker, Sebastian Bodenstedt, Christian Stock, Eric Heim, Michael Götz, S Wirkert, Hannes Kenngott, 2016. Crowd-algorithm collaboration for large-scale endoscopic image annotation with confidence. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19. Springer, 616–623.
Raphaël Marée, Benjamin Stévens, Loïc Rollus, Natacha Rocks, Xavier Moles Lopez, Isabelle Salmon, Didier Cataldo, and Louis Wehenkel. 2013. A rich internet application for remote visualization and collaborative annotation of digital slides in histology and cytology. In Diagnostic Pathology, Vol. 8. BioMed Central, 1–4.
Abir Masmoudi, Emna Mezghani, Hatem Bellaaj, Khalil Drira, and Mohamed Jmaiel. 2017. A web-based knowledge management system for scientific research team. In 2017 IEEE 26th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE). IEEE, 296–301.
Lucas McCullum, Hasan Saeed, Benjamin Moody, Diane Perry, Eric Gottlieb, Tom Pollard, Xavier Borrat Frigola, Qiao Li, Gari Clifford, Roger G Mark, . An Open-Source Platform for Collaborative Annotation of Physiological Waveforms. ( ), .
Marcio Ferreira Moreno, Wallas Henrique Sousa Dos Santos, Rodrigo Costa Mesquita Santos, and Renato Fontoura De Gusmao Cerqueira. 2018. Supporting knowledge creation through has: The hyperknowledge annotation system. In 2018 IEEE International Symposium on Multimedia (ISM). IEEE, 239–246.
Julie A Murphy. 2021. Collaborative Annotation: Tools for Enhancing Learning and Scholarly Communication. Serials Review 47, 3-4 (2021), 157–162.
Steven R Ness, Alex Lerch, and George Tzanetakis. 2011. Strategies for orca call retrieval to support collaborative annotation of a large archive. In 2011 IEEE 13th International Workshop on Multimedia Signal Processing. IEEE, 1–6.
Mariana Neves and Jurica Ševa. 2021. An extensive review of tools for manual annotation of documents. Briefings in bioinformatics 22, 1 (2021), 146–163.
Lucas ES Oliveira, Caroline P Gebeluca, Adalniza MP Silva, Claudia MC Moro, Sadid A Hasan, and Oladimeji Farri. 2017. A statistics and UMLS-based tool for assisted semantic annotation of Brazilian clinical documents. In 2017 IEEE International conference on bioinformatics and biomedicine (BIBM). IEEE, 1072–1078.
Yuxin Pan, Jianwu Fang, Jian Dou, Zhen Ye, and Jianru Xue. 2018. Research on Human-Machine Collaborative Annotation for Traffic Scene Data. In 2018 Chinese Automation Congress (CAC). IEEE, 2900–2905.
Frederic Papazian, R. Robert Bossy, and Claire Nédellec. 2012. AlvisAE: a collaborative Web text annotation editor for knowledge acquisition. In LAW VI’12 - Sixth Linguistic Annotation Workshop.
Vishwajeet Pattanaik, Shweta Suran, and Dirk Draheim. 2019. Enabling social information exchange via dynamically robust annotations. In Proceedings of the 21st International Conference on Information Integration and Web-based Applications & Services. 176–184.
Martín Pérez-Pérez, Daniel Glez-Peña, Florentino Fdez-Riverola, and Anália Lourenço. 2015. Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects. Computer methods and programs in biomedicine 118, 2 (2015), 242–251.
Johann Poignant, Mateusz Budnik, Hervé Bredin, Claude Barras, Mickael Stefas, Pierrick Bruneau, Gilles Adda, Laurent Besacier, Hazim Ekenel, Gil Francopoulo, 2016. The CAMOMILE collaborative annotation platform for multi-modal, multi-lingual and multi-media documents. In LREC 2016 Conference.
Elena Potapenko, Pascal Kovacs, and Elisabeth Burr. 2013. “From Leipzig into the Romania”-Environment for collaborative annotation and knowledge creation. In Proceedings of the 1st International Workshop on Collaborative Annotations in Shared Environment: metadata, vocabularies and techniques in the Digital Humanities. 1–4.
Daniel Racoceanu and Frédérique Capron. 2015. Towards semantic-driven high-content image analysis: An operational instantiation for mitosis detection in digital histopathology. Computerized Medical Imaging and Graphics 42 (2015), 2–15.
Lozana Rossenova, Zoe Schubert, Richard Vock, Lucia Sohmen, Lukas Günther, Paul Duchesne, and Ina Blümel. 2022. Collaborative annotation and semantic enrichment of 3D media: a FOSS toolchain. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries. 1–5.
Christopher Rydell and Joakim Lindblad. 2021. CytoBrowser: a browser-based collaborative annotation platform for whole slide images. F1000Research 10 (03 2021), 226. DOI: 10.12688/f1000research.51916.1
Aziret Satybaldiev, Peter Hevesi, Marco Hirsch, Vitor Fortes Rey, and Paul Lukowicz. 2019. CoAT: a web-based, collaborative annotation tool. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. 814–818.
Marinette Savonnet, Eric Leclercq, and Pierre Naubourg. 2015. eclims: An extensible and dynamic integration framework for biomedical information systems. IEEE journal of biomedical and health informatics 20, 6 (2015), 1640–1649.
Roger Schaer, Sebastian Otálora, Oscar Jimenez-del Toro, Manfredo Atzori, and Henning Müller. 2019. Deep learning-based retrieval system for gigapixel histopathology cases and the open access literature. Journal of pathology informatics 10, 1 (2019), 19.
Kleberson A Serique, Alan Snyder, Debra Willrett, Daniel L Rubin, and Dilvan A Moreira. 2012. Using the semantic web and web apps to connect radiologists and oncologists. In 2012 IEEE 21st International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. IEEE, 480–485.
Hal Shin, Natanael Exe, Ujwal Dutta, Tanka Raj Joshi, Sagar Verma, and Siddharth Gupta. 2022. Europa: Increasing Accessibility of Geospatial Datasets. In IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium. 6848–6851. DOI: 10.1109/IGARSS46834.2022.9884440
Ehsan Jahangirzadeh Soure, Emily Kuang, Mingming Fan, and Jian Zhao. 2021. CoUX: collaborative visual analysis of think-aloud usability test videos for digital interfaces. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 643–653.
Addison YS Su, Stephen JH Yang, Wu-Yuin Hwang, and Jia Zhang. 2010. A Web 2.0-based collaborative annotation system for enhancing knowledge sharing in collaborative learning environments. Computers & Education 55, 2 (2010), 752–766.
Dan Su and Pascale Fung. 2012. Personalized music emotion classification via active learning. In Proceedings of the second international ACM workshop on Music information retrieval with user-centered and multimodal strategies. 57–62.
Jaana Takis, AQM Saiful Islam, Christoph Lange, and Sören Auer. 2015. Crowdsourced semantic annotation of scientific publications and tabular data in PDF. In Proceedings of the 11th International Conference on Semantic Systems. 1–8.
Nikolaos Tsipas, Panagiotis Zapartas, Lazaros Vrysis, and Charalampos Dimoulas. 2015. Augmenting social multimedia semantic interaction through audio-enhanced web-tv services. In Proceedings of the Audio Mostly 2015 on Interaction With Sound. 1–7.
Jasper RR Uijlings, Mykhaylo Andriluka, and Vittorio Ferrari. 2020. Panoptic Image Annotation with a Collaborative Assistant. In Proceedings of the 28th ACM International Conference on Multimedia. 3302–3310.
Victoria Uren, Philipp Cimiano, José Iria, Siegfried Handschuh, Maria Vargas-Vera, Enrico Motta, and Fabio Ciravegna. 2006. Semantic annotation for knowledge management: Requirements and a survey of the state of the art. Journal of Web Semantics 4, 1 (2006), 14–28.
Byron C Wallace, Kevin Small, Carla E Brodley, Joseph Lau, and Thomas A Trikalinos. 2012. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT international health informatics symposium. 819–824.
Haiyan Wang, Yanjie Zhou, Bing Zhou, and Zongmin Wang. 2021. A Novel Method for Detection of ECG with Deep Learning. In 2021 7th International Conference on Computer and Communications (ICCC). IEEE, 631–635.
Jessica Z Wang, Amy X Zhang, and David R Karger. 2022. Designing for Engaging with News using Moral Framing towards Bridging Ideological Divides. Proceedings of the ACM on Human-Computer Interaction 6, GROUP (2022), 1–23.
Nicholas Warren, Benjamin Garrard, Elliot Staudt, and Jelena Tesic. 2018. Transfer learning of deep neural networks for visual collaborative maritime asset identification. In 2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC). IEEE, 246–255.
Antoine Widlöcher and Yann Mathet. 2012. The glozz platform: A corpus annotation and mining tool. In Proceedings of the 2012 ACM symposium on Document engineering. 171–180.
Stefan Wilk, Stephan Kopf, and Wolfgang Effelsberg. 2016. Collaborative annotation of videos relying on weak consistency. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 12, 3 (2016), 1–23.
Jan Witowski, Jongmun Choi, Soomin Jeon, Doyun Kim, Joowon Chung, John Conklin, Maria Gabriela Figueiro Longo, Marc D Succi, and Synho Do. 2021. MarkIt: a collaborative artificial intelligence annotation platform leveraging blockchain for medical imaging research. Blockchain in Healthcare Today (2021).
Joanna Wolfe. 2002. Annotation technologies: A software and research review. Computers and Composition 19, 4 (2002), 471–497.
Jialun Wu, Anyu Mao, Xinrui Bao, Haichuan Zhang, Zeyu Gao, Chunbao Wang, Tieliang Gong, and Chen Li. 2021. Pimip: An open source platform for pathology information management and integration. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2088–2095.
Xuehong Wu, Junwen Duan, Jianhua Li, and Min Li. 2021. nPTAS: A Novel Platform for Text Annotation and Service. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 544–549.
Zhicheng Yang, Zhenjie Cao, Yanbo Zhang, Peng Chang, Shibin Wu, Lingyun Huang, Wei Xu, Mei Han, Jing Xiao, Mingxiang Wu, 2021. MABEL: An AI-powered mammographic breast lesion diagnostic system. In 2020 IEEE International Conference on E-health Networking, Application & Services (HEALTHCOM). IEEE, 1–7.
Chih-Hao Yu, Tudor Groza, and Jane Hunter. 2013. Reasoning on crowd-sourced semantic annotations to facilitate cataloguing of 3D artefacts in the cultural heritage domain. In The Semantic Web–ISWC 2013: 12th International Semantic Web Conference, Sydney, NSW, Australia, October 21-25, 2013, Proceedings, Part II 12. Springer, 228–243.
Jiaxin Yu, Florian Wellmann, Simon Virgo, Marven von Domarus, Mingze Jiang, Joyce Schmatz, and Bastian Leibe. 2023. Superpixel segmentations for thin sections: evaluation of methods to enable the generation of machine learning training data sets. Computers & Geosciences 170 (2023), 105232.
Beste F Yuksel, Pooyan Fazli, Umang Mathur, Vaishali Bisht, Soo Jung Kim, Joshua Junhee Lee, Seung Jung Jin, Yue-Ting Siu, Joshua A Miele, and Ilmi Yoon. 2020. Human-in-the-loop machine learning to increase video accessibility for visually impaired and blind users. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 47–60.
Lishi Zhang, Chenghan Fu, and Jia Li. 2018. Collaborative annotation of semantic objects in images with multi-granularity supervisions. In Proceedings of the 26th ACM international conference on Multimedia. 474–482.
Christos Zigkolis, Symeon Papadopoulos, George Filippou, Yiannis Kompatsiaris, and Athena Vakali. 2014. Collaborative event annotation in tagged photo collections. Multimedia Tools and Applications 70 (2014), 89–118.
Tobias Zimmermann, Markus Weber, Marcus Liwicki, and Didier Stricker. 2012. CoVidA: pen-based collaborative video annotation. In Proceedings of the 1st International Workshop on Visual Interfaces for Ground Truth Collection in Computer Vision Applications. 1–6.
Damien E Zomahoun and Kokou Yetongnon. 2014. EMERGSEM: emergent semantic and recommendation system for image retrieval. In 2014 Tenth International Conference on Signal-Image Technology and Internet-Based Systems. IEEE, 256–263.
Publicado
16/10/2023
Como Citar
RIBEIRO DE SOUZA, Joana Gabriela; JANSEN FERREIRA, Juliana; SEGURA, Vinicius.
A Taxonomy of Methods, Tools, and Approaches for Enabling Collaborative Annotation. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 22. , 2023, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
