Towards LLM-Based Detection of Elixir Code Smells: An Empirical Evaluation

• Mateus José Dias UFV
• Lucas Vegi UFV

DOI: https://doi.org/10.5753/se4fp.2026.32884

Resumo

Large Language Models (LLMs) have shown promise for code smell detection, but their effectiveness for functional programming languages remains largely unexplored. This paper presents the first empirical evaluation of LLM-based Elixir code smell identification. We introduce ExSmell-Train, a retrieval corpus, ExSmell-Gold, a labeled benchmark, and ExSmellBench, an experimental framework for evaluating four prompting strategies across the 35 smells of the Elixir smell catalog. We compare one proprietary and three openweight LLMs under description-based and Retrieval-Augmented Generation (RAG) prompting. The results show that RAG improves multi-label smell identification only when combined with smell descriptions, whereas retrieved examples alone are insufficient. Performance varies substantially across smells, categories, and models: design-related Elixir smells are identified more reliably than traditional smells, and no model dominates across all criteria. Gemini 3.1 Flash Lite achieves the highest recall, whereas Qwen2.5-Coder 32B provides the best precision and overall agreement. Our publicly available datasets and benchmark establish a foundation for future research on LLM-based smell detection for functional languages.

Palavras-chave: Code Smells, Elixir, Functional Programming, AI4SE

Referências

Maria Christakis and Christian Bird. 2016. What developers want and need from program analysis: an empirical study. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore) (ASE ’16). Association for Computing Machinery, New York, NY, USA, 332–343. DOI: 10.1145/2970276.2970347

Mateus José Dias and Lucas Francisco da Matta Vegi. 2026. ExSmell-Gold: Gold Standard Dataset. [link].

Mateus José Dias and Lucas Francisco da Matta Vegi. 2026. ExSmell-Train: Dataset and Experimental Artifacts. [link].

Mateus José Dias and Lucas Francisco da Matta Vegi. 2026. ExSmellBench: A Benchmarking Tool for Evaluating LLM-Based Elixir Code Smell Detection. [link].

Martin Fowler and Kent Beck. 1999. Refactoring: improving the design of existing code (1 ed.). Addison-Wesley.

Qi Guo, Xiaofei Xie, Shangqing Liu, Ming Hu, Xiaohong Li, and Lei Bu. 2025. Intention is All You Need: Refining Your Code from Your Intention. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE ’25). IEEE Press, 1127–1139. DOI: 10.1109/ICSE55347.2025.00191

Saša Jurić. 2024. Elixir in action (3 ed.). Manning.

Mahmoud Kassem, Francisco Ribeiro, and Sarah Nadi. 2026. An Automated Methodology for Generating Labeled Datasets of Semantic Errors in Code. In Proceedings of the 3rd International Workshop on Large Language Models For Code (LLM4Code ’26). Association for Computing Machinery, New York, NY, USA, 14–20. DOI: 10.1145/3786181.3788730

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. DOI: 10.1162/tacl_a_00638

Luciana Lourdes Silva, Jânio Silva, João Eduardo Montandon, Marcus Andrade, and Marco Tulio Valente. 2024. Detecting Code Smells using ChatGPT: Initial Insights. In 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–7.

Elder Sobrinho, Andrea De Lucia, and Marcelo Maia. 2021. A systematic literature review on bad smells–5 w’s: which, when, what, who, where. IEEE Transactions on Software Engineering 47, 1 (2021), 17–66. DOI: 10.1109/TSE.2018.2880977

Lucas Francisco da Matta Vegi and Marco Tulio Valente. 2023. Understanding code smells in Elixir functional language. Empirical Software Engineering 28, 102 (2023), 1–32. DOI: 10.1007/s10664-023-10343-6