Analysis of AI Detection Tools for Scientific Texts in Portuguese Generated by ChatGPT, Gemini, and DeepSeek

• Lucas S. Candido IFAL
• Christian A. de Melo Barbosa IFAL
• Lylia G. Martins IFAL
• Esdras J. H. Costa IFAL

DOI: https://doi.org/10.5753/wics.2025.8692

Abstract

This study investigated the effectiveness of five AI detection tools (ZeroGPT, JustDone AI Detector, Writer AI Detector, Seo AI Detector and Summarizer AI Detector) in identifying scientific texts in Portuguese generated by different language models (ChatGPT, Gemini and DeepSeek), comparing them with control samples written by humans. The research used 50 manuscripts and error metrics (MAE and RMSE) to evaluate the performance of the detectors. The results showed that ZeroGPT had the highest accuracy in detecting AI-generated content, with the lowest average errors for synthetic texts. However, even ZeroGPT had a false positive rate when classifying human text. The other tools had limitations such as high false positive rates, low sensitivity in detecting AI in Portuguese, or inconsistency in results. The analysis concludes that although ZeroGPT is the most effective tool among those evaluated, the lack of a perfect solution underscores the need for continuous advances in AI detection technology to ensure the integrity of scientific production.

References

Almeida, F. das C. F., Aguiar, Y. P. C. and Magalhaes, R. M. (16 oct 2023). Você sabe diferenciar um resumo escrito por humanos do gerado pelo ChatGPT? In Workshop sobre Aspectos da Interação Humano-Computador na Web Social (WAIHCWS). . SBC. [link], [accessed on Mar 20].

Cabanac, G. and Labbé, C. (2021). Prevalence of nonsensical algorithmically generated papers in the scientific literature. Journal of the Association for Information Science and Technology, v. 72, n. 12, p. 1461–1476.

Candido, L. S., Barbosa, C. A. de M. and Costa, E. J. H. (21 jul 2024). Análise de ferramentas para detecção de textos científicos gerados por Inteligência Artificial (ChatGPT). In Workshop sobre as Implicações da Computação na Sociedade (WICS). . SBC. [link], [accessed on Mar 22].

Chen, B., Zhang, Z., Langrené, N. and Zhu, S. (5 sep 2024). Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review. . arXiv. [link], [accessed on Dec 30].

DeepSeek-AI, Liu, A., Feng, B., et al. (18 feb 2025). DeepSeek-V3 Technical Report. . arXiv. [link], [accessed on Mar 13].

Elkhatat, A. M., Elsaid, K. and Almeer, S. (1 sep 2023). Evaluating the efficacy of AI content detection tools in differentiating between human and AI-generated text. International Journal for Educational Integrity, v. 19, n. 1, p. 17.

Gil, A. C. (2017). Como elaborar projetos de pesquisa. 6. ed ed. São Paulo: Atlas.

Google DeepMind (12 mar 2025). Gemini 2.0. [link], [accessed on Mar 13].

Hammad, M. (1 mar 2023). The Impact of Artificial Intelligence (AI) Programs on Writing Scientific Research. Annals of Biomedical Engineering, v. 51, n. 3, p. 459–460.

Intelligent.com (23 jan 2023). Nearly 1 in 3 College Students Have Used ChatGPT on Written Assignments. [link], [accessed on Mar 19].

JustDone (2025). AI Detector | JustDone. [link], [accessed on Mar 18].

Ladha, N., Yadav, K. and Rathore, P. (25 oct 2023). AI-Generated Content Detectors: Boon or Bane for Scientific Writing. Indian Journal of Science and Technology, v. 16, n. 39, p. 3435–3439.

Liyanage, V., Buscaldi, D. and Nazarenko, A. (jun 2022). A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications. [N. Calzolari, F. Béchet, P. Blache, et al., Eds.]In Proceedings of the Thirteenth Language Resources and Evaluation Conference. . European Language Resources Association. [link], [accessed on Mar 17].

Malik, M. A. and Amjad, A. I. (12 jan 2025). AI vs AI: How effective are Turnitin, ZeroGPT, GPTZero, and Writer AI in detecting text generated by ChatGPT, Perplexity, and Gemini? Journal of Applied Learning and Teaching, v. 8, n. 1.

OpenAI (18 jul 2024). GPT-4o mini: advancing cost-efficient intelligence. [link], [accessed on Dec 23].

Seo.Ai (2024). Free AI Content Detector. [link], [accessed on Mar 18].

Sullivan, M., Kelly, A. and McLaughlan, P. (20 mar 2023). ChatGPT in higher education: Considerations for academic integrity and student learning. Journal of Applied Learning and Teaching, v. 6, n. 1, p. 31–40.

Summarizer.org (2025). Free AI Detector. [link], [accessed on Mar 18].

Team, G., Georgiev, P., Lei, V. I., et al. (16 dec 2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. . arXiv. [link], [accessed on Mar 13].

Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., et al. (25 dec 2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, v. 19, n. 1, p. 26.

Writer (jan 2024). Writer AI content detector disclaimer. [link], [accessed on Mar 23].

Writer (2025). AI content detector. [link], [accessed on Mar 18].

ZeroGPT.net (2025). ZeroGPT - Detector de Chat GPT | Verificador de Chat GPT. [link], [accessed on Mar 18].