Test Smells through the Lens of the Dart Community: A Mixed-Methods Study on Grey Literature and Developer Perceptions
Resumo
Automated testing is a cornerstone of modern software quality, especially in rapidly growing ecosystems like Dart and Flutter. However, poor design choices in tests, known as test smells, can compromise maintainability and reliability. While test smells are widely studied in Java and C#, their prevalence and developer perceptions in the Dart ecosystem remain largely unexplored. This work aims to investigate how Dart developers perceive bad testing practices and identify which test smell patterns emerge from real-world community discussions.We conducted a mixed-methods study on grey literature (specifically Stack Overflow Q&A posts), analyzing 307 selected discussions that yielded 432 individual test smell occurrences. Applying Pareto Analysis, we find that a small subset of seven smells (the "Vital Few") accounts for approximately 80% of all occurrences, with Magic Number Test, Unknown Test, and Assertion Roulette being the most prevalent. Furthermore, we identify and formalize the Expected Resolution Omission (ERO) smell, a Dart-specific asynchronous testing anti-pattern that causes silent false positives or flaky tests. Our findings highlight critical pain points in the Dart testing community and provide a formally defined catalog to support developers and tool builders in improving test quality.
Palavras-chave:
Automated Testing, Test Smells, Dart, Stack Overflow, Pareto Analysis
Referências
Mumtahina Ahmed, Md Nahidul Islam Opu, Chanchal Roy, Sujana Islam Suhi, and Shaiful Chowdhury. 2026. Exploring challenges in test mocking: Developer questions and insights from StackOverflow. Journal of Systems and Software 235 (2026), 112748. DOI: 10.1016/j.jss.2025.112748
Wajdi Aljedaani, Anthony Peruma, Ahmed Aljohani, Mazen Alotaibi, Mohamed Wiem Mkaouer, Ali Ouni, Christian D Newman, Abdullatif Ghallab, and Stephanie Ludi. 2021. Test smell detection tools: A systematic mapping study. In Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering. 170–180. DOI: 10.1145/3463274.3463335
Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave Binkley. 2015. Are test smells really harmful? an empirical study. Empirical Software Engineering 20 (2015), 1052–1094. DOI: 10.1007/s10664-014-9313-0
Humberto Damasceno, Carla Bezerra, Denivan Campos, Ivan Machado, and Emanuel Coutinho. 2023. Test smell refactoring revisited: What can internal quality attributes and developers’ experience tell us? Journal of Software Engineering Research and Development 11, 1 (2023), 13–1. DOI: 10.5753/jserd.2023.3195
Vahid Garousi and Barış Küçük. 2018. Smells in software test code: A survey of knowledge in industry and academia. Journal of systems and software 138 (2018), 52–81. DOI: 10.1016/j.jss.2017.12.013
Vahid Garousi, Baris Kucuk, and Michael Felderer. 2018. What we know about smells in software test code. IEEE Software 36, 3 (2018), 61–73. DOI: 10.1109/MS.2018.2875843
Google. 2025. Dart: Language Tour. [link]. Acesso em 13 jul. 2025.
Google. 2025. flutter_test API Reference. [link]. Acesso em 13 jul. 2025.
Google. 2025. package:test. [link]. Acesso em 13 jul. 2025.
Nildo Silva Junior, Luana Martins, Larissa Rocha, Heitor Costa, and Ivan Machado. 2021. How are test smells treated in the wild? A tale of two empirical studies. Journal of Software Engineering Research and Development 9 (2021), 9–1. DOI: 10.5753/jserd.2021.1802
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174. DOI: 10.2307/2529310
Luana Martins, Heitor Costa, and Ivan Machado. 2024. On the diffusion of test smells and their relationship with test code quality of Java projects. Journal of Software: Evolution and Process 36, 4 (2024), e2532. DOI: 10.1002/smr.2532
Annibale Panichella, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent J. Hellendoorn. 2022. Test smells 20 years later: detectability, validity, and reliability. Empirical Softw. Engg. 27, 7 (Dec. 2022), 40 pages. DOI: 10.1007/s10664-022-10207-5
Anthony Peruma, Khalid Almalki, Christian D Newman, Mohamed Wiem Mkaouer, Ali Ouni, and Fabio Palomba. 2020. Tsdetect: An open source test smells detection tool. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 1650–1654. DOI: 10.1145/3368089.3417921
Valeria Pontillo, Luana Martins, Ivan Machado, Fabio Palomba, and Filomena Ferrucci. 2025. An empirical investigation into the capabilities of anomaly detection approaches for test smell detection. Journal of Systems and Software 222 (2025), 112320. DOI: 10.1016/j.jss.2024.112320
Railana Santana, Luana Martins, Tássio Virgínio, Larissa Rocha, Heitor Costa, and Ivan Machado. 2024. An empirical evaluation of RAIDE: A semi-automated approach for test smells detection and refactoring. Science of Computer Programming 231 (2024), 103013. DOI: 10.1016/j.scico.2023.103013
Tushar Sharma, Pratibha Mishra, and Rohit Tiwari. 2016. Designite: a software design quality assessment tool. In Proceedings of the 1st InternationalWorkshop on Bringing Architectural Design Thinking into Developers’ Daily Activities (Austin, Texas) (BRIDGE ’16). Association for Computing Machinery, New York, NY, USA, 1–4. DOI: 10.1145/2896935.2896938
Davide Spadini, Fabio Palomba, Andy Zaidman, Magiel Bruntink, and Alberto Bacchelli. 2018. On the relation of test smells to software code quality. In 2018 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 1–12. DOI: 10.1109/ICSME.2018.00010
Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2016. An empirical investigation into the nature of test smells. In Proceedings of the 31st IEEE/ACMinternational conference on automated software engineering. 4–15. DOI: 10.1145/2970276.2970340
Lionel Sujay Vailshery. 2025. Cross-platform mobile frameworks used by developers globally 2019-2023. [link] Accessed: 2025-07-14.
Arie Van Deursen, Leon Moonen, Alex Van Den Bergh, and Gerard Kok. 2001. Refactoring test code. In Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001), Vol. 92. Citeseer, 95.
Tássio Virgínio, Luana Martins, Railana Santana, Adriana Cruz, Larissa Rocha, Heitor Costa, and Ivan Machado. 2021. On the test smells detection: an empirical study on the jnose test accuracy. Journal of Software Engineering Research and Development 9 (2021), 8–1. DOI: 10.5753/jserd.2021.1893
Tássio Virgínio, Marcio Ribeiro, and Ivan Machado. 2025. DNose: Dart Test Smell Detector. In Anais do XXXIX Simpósio Brasileiro de Engenharia de Software (Recife/PE). SBC, Porto Alegre, RS, Brasil, 976–982. DOI: 10.5753/sbes.2025.11564
Tássio Virgínio, Márcio Ribeiro, and Ivan Machado. 2025. On the prevalence of test smells in mobile development. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 84–93. DOI: 10.5753/sast.2025.14327
Jing Wang, Weixi Zhang, Weiwei Wang, Ruilian Zhao, and Ying Shang. 2025. Predicting the Root Cause of Flaky Tests Based on Test Smells. In 2025 IEEE/ACM 22nd International Conference on Software and Systems Reuse (ICSR) (Ottawa, ON, Canada). IEEE Press, 84–94. DOI: 10.1109/ICSR66718.2025.00015
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, Anders Wesslén, et al. 2012. Experimentation in software engineering. Vol. 236. Springer. DOI: 10.1007/978-3-642-29044-2
Min Zhang, Tracy Hall, and Nathan Baddoo. 2011. Code bad smells: a review of current knowledge. Journal of Software Maintenance and Evolution: research and practice 23, 3 (2011), 179–202. DOI: 10.1002/smr.521
Wajdi Aljedaani, Anthony Peruma, Ahmed Aljohani, Mazen Alotaibi, Mohamed Wiem Mkaouer, Ali Ouni, Christian D Newman, Abdullatif Ghallab, and Stephanie Ludi. 2021. Test smell detection tools: A systematic mapping study. In Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering. 170–180. DOI: 10.1145/3463274.3463335
Gabriele Bavota, Abdallah Qusef, Rocco Oliveto, Andrea De Lucia, and Dave Binkley. 2015. Are test smells really harmful? an empirical study. Empirical Software Engineering 20 (2015), 1052–1094. DOI: 10.1007/s10664-014-9313-0
Humberto Damasceno, Carla Bezerra, Denivan Campos, Ivan Machado, and Emanuel Coutinho. 2023. Test smell refactoring revisited: What can internal quality attributes and developers’ experience tell us? Journal of Software Engineering Research and Development 11, 1 (2023), 13–1. DOI: 10.5753/jserd.2023.3195
Vahid Garousi and Barış Küçük. 2018. Smells in software test code: A survey of knowledge in industry and academia. Journal of systems and software 138 (2018), 52–81. DOI: 10.1016/j.jss.2017.12.013
Vahid Garousi, Baris Kucuk, and Michael Felderer. 2018. What we know about smells in software test code. IEEE Software 36, 3 (2018), 61–73. DOI: 10.1109/MS.2018.2875843
Google. 2025. Dart: Language Tour. [link]. Acesso em 13 jul. 2025.
Google. 2025. flutter_test API Reference. [link]. Acesso em 13 jul. 2025.
Google. 2025. package:test. [link]. Acesso em 13 jul. 2025.
Nildo Silva Junior, Luana Martins, Larissa Rocha, Heitor Costa, and Ivan Machado. 2021. How are test smells treated in the wild? A tale of two empirical studies. Journal of Software Engineering Research and Development 9 (2021), 9–1. DOI: 10.5753/jserd.2021.1802
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174. DOI: 10.2307/2529310
Luana Martins, Heitor Costa, and Ivan Machado. 2024. On the diffusion of test smells and their relationship with test code quality of Java projects. Journal of Software: Evolution and Process 36, 4 (2024), e2532. DOI: 10.1002/smr.2532
Annibale Panichella, Sebastiano Panichella, Gordon Fraser, Anand Ashok Sawant, and Vincent J. Hellendoorn. 2022. Test smells 20 years later: detectability, validity, and reliability. Empirical Softw. Engg. 27, 7 (Dec. 2022), 40 pages. DOI: 10.1007/s10664-022-10207-5
Anthony Peruma, Khalid Almalki, Christian D Newman, Mohamed Wiem Mkaouer, Ali Ouni, and Fabio Palomba. 2020. Tsdetect: An open source test smells detection tool. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 1650–1654. DOI: 10.1145/3368089.3417921
Valeria Pontillo, Luana Martins, Ivan Machado, Fabio Palomba, and Filomena Ferrucci. 2025. An empirical investigation into the capabilities of anomaly detection approaches for test smell detection. Journal of Systems and Software 222 (2025), 112320. DOI: 10.1016/j.jss.2024.112320
Railana Santana, Luana Martins, Tássio Virgínio, Larissa Rocha, Heitor Costa, and Ivan Machado. 2024. An empirical evaluation of RAIDE: A semi-automated approach for test smells detection and refactoring. Science of Computer Programming 231 (2024), 103013. DOI: 10.1016/j.scico.2023.103013
Tushar Sharma, Pratibha Mishra, and Rohit Tiwari. 2016. Designite: a software design quality assessment tool. In Proceedings of the 1st InternationalWorkshop on Bringing Architectural Design Thinking into Developers’ Daily Activities (Austin, Texas) (BRIDGE ’16). Association for Computing Machinery, New York, NY, USA, 1–4. DOI: 10.1145/2896935.2896938
Davide Spadini, Fabio Palomba, Andy Zaidman, Magiel Bruntink, and Alberto Bacchelli. 2018. On the relation of test smells to software code quality. In 2018 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 1–12. DOI: 10.1109/ICSME.2018.00010
Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2016. An empirical investigation into the nature of test smells. In Proceedings of the 31st IEEE/ACMinternational conference on automated software engineering. 4–15. DOI: 10.1145/2970276.2970340
Lionel Sujay Vailshery. 2025. Cross-platform mobile frameworks used by developers globally 2019-2023. [link] Accessed: 2025-07-14.
Arie Van Deursen, Leon Moonen, Alex Van Den Bergh, and Gerard Kok. 2001. Refactoring test code. In Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001), Vol. 92. Citeseer, 95.
Tássio Virgínio, Luana Martins, Railana Santana, Adriana Cruz, Larissa Rocha, Heitor Costa, and Ivan Machado. 2021. On the test smells detection: an empirical study on the jnose test accuracy. Journal of Software Engineering Research and Development 9 (2021), 8–1. DOI: 10.5753/jserd.2021.1893
Tássio Virgínio, Marcio Ribeiro, and Ivan Machado. 2025. DNose: Dart Test Smell Detector. In Anais do XXXIX Simpósio Brasileiro de Engenharia de Software (Recife/PE). SBC, Porto Alegre, RS, Brasil, 976–982. DOI: 10.5753/sbes.2025.11564
Tássio Virgínio, Márcio Ribeiro, and Ivan Machado. 2025. On the prevalence of test smells in mobile development. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 84–93. DOI: 10.5753/sast.2025.14327
Jing Wang, Weixi Zhang, Weiwei Wang, Ruilian Zhao, and Ying Shang. 2025. Predicting the Root Cause of Flaky Tests Based on Test Smells. In 2025 IEEE/ACM 22nd International Conference on Software and Systems Reuse (ICSR) (Ottawa, ON, Canada). IEEE Press, 84–94. DOI: 10.1109/ICSR66718.2025.00015
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, Anders Wesslén, et al. 2012. Experimentation in software engineering. Vol. 236. Springer. DOI: 10.1007/978-3-642-29044-2
Min Zhang, Tracy Hall, and Nathan Baddoo. 2011. Code bad smells: a review of current knowledge. Journal of Software Maintenance and Evolution: research and practice 23, 3 (2011), 179–202. DOI: 10.1002/smr.521
Publicado
08/09/2026
Como Citar
JUNIOR, Eronildo; ROCHA, Rafael; PASSOS, Paloma; CERQUEIRA, Paulo; VIRGÍNIO, Tássio; SILVA, Antonio Rito; MACHADO, Ivan.
Test Smells through the Lens of the Dart Community: A Mixed-Methods Study on Grey Literature and Developer Perceptions. In: SIMPÓSIO BRASILEIRO DE TESTES DE SOFTWARE SISTEMÁTICO E AUTOMATIZADO (SAST), 11. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 29-38.
DOI: https://doi.org/10.5753/sast.2026.30753.
