Analysis of tools for detecting scientific texts generated by Artificial Intelligence (ChatGPT)

• Lucas S. Candido IFAL
• Christian A. de Melo Barbosa IFAL
• Esdras J. H. Costa IFAL

DOI: https://doi.org/10.5753/wics.2024.2929

Abstract

This article analyzes the performance of four tools for detecting articles generated by artificial intelligence (AI), Writer AI Content Detector, GPT Zero, Zero GPT and Quillbot AI Content Detector. The detectors are evaluated on a dataset of 30 articles generated in Portuguese on ChatGPT 3.5 and 10 human-written text. The results obtained from the experiment show the effectiveness of Zero GPT in categorizing content produced by machines, achieving an accuracy of 97.50%, a performance comparable to the state-of the-art. However, challenges were encountered in identifying artificially generated texts, and the research suggests the need for continuous improvements and explorations to improve the process of identifying artificially generated content.

References

Generating sentiment-preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-based Detection. Adelani, David Ifeoluwa, et al. 2020. 2020. Advanced Information Networking and Applications.

A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications. Vijini, Liyanage, et al. 2022. 2022. Proceedings of the Thirteenth Language Resources and Evaluation Conference.

Cabanac, Guillaume e Labbé, Cyrill. 2021. Prevalence of nonsensical algorithmically generated papersin the scientific literature. JASIST. Dezembro de 2021, pp. 1461-1476.

Chaka, Chaka e al, et. 2023. Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools. Journal of Applied Learning & Teaching. 2023.

Deepfake Text Detection: Limitations and Opportunities. Jiameng, Pu, Zain, Sarwar e Sifat, Muhammad Abdullah. 2023. San Francisco : s.n., 2023. IEEE Symposium on Security and Privacy .

Dehouche, N. 2021. Plagiarism in the Age of Massive Generative Pre-Trained Transformers (Gpt-3). Ethics in Science and Environmental Politics. 2021, pp. 17–23.

Elkhatat, A.M., Elsaid, K e Almeer, S. 2023. Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text. International Journal for Educational Integrity. 2023.

FERREIRA, Rafael Clementino Veríssimo, GARCIA, Gustavo Henrique Maia e BRASIL, Deilton Ribeiro. 2023. O surgimento do Chat GPT e a insegurança sobre o futuro dos trabalhos acadêmicos. Cadernos de Dereito Actual. 2023, pp. 130-143.

Gil, Antonio Carlos. 2017. Como Elaborar Projetos de Pesquisa. 6. São Paulo : Atlas, 2017.

GPTZero. 2023. GPTZero: The Global Standard for AI Detection, Humans Deserve the Truth. [Online] GPTZero, 2023. [Citado em: 06 de 30 de 2023.] [link].

IGA: An Intent-Guided Authoring Assistant. Sun, Simeng, Zhao, Wenlong e Manjunatha, Varun. 2021. 2021. Empirical Methods in Natural Language Processing.

Kıyak, Yavuz Selim. 2023. A ChatGPT Prompt for Writing Case-Based Multiple-Choice Questions. Revista Española de Educación Médica. 23 de 10 de 2023, pp. 98-103.

Language Models are Few-Shot Learners. BROWN, Tom e al, et. 2020. Vancouver : s.n., 2020. Advances in neural information processing systems. pp. 1877-1901.

QuillBot. 2017. QuillBot AI Detector. QuillBot. [Online] 14.957.4, Learneo, 2017. [Citado em: 12 de 12 de 2023.] [link].

Stokel-Walker, C. AI bot ChatGPT writes smart essays-should academics worry? Nature.

Weber Wulff, Debora e al, et. 2023. Testing of detection tools for AI generated text. International Journal for Educational Integrity. 2023.

White, Jules e al., et. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. Arxiv. 21 de 2 de 2023.

WRITER. 2024. AI Content Detector. Writer. [Online] Writer, 2024. [Citado em: 04 de 03 de 2024.] [link].

ZeroGPT. 2023. AI Detector - Trusted AI Checker for ChatGPT, GPT4 & Bard. ZeroGPT. [Online] ZeroGPT, 01 de 01 de 2023. [Citado em: 12 de 12 de 2023.] [link].