Test-Driven Prompting: An Empirical Evaluation of Methods Reconstruction based on Unit Tests
Resumo
Unit tests are a fundamental component of modern software development. Besides their role in verifying the correctness of code, developers also rely on them to document and specify the expected behavior of methods and functions. We argue that Large Language Models (LLMs) can also benefit from them to generate correct source code. This paper investigates to what extent unit tests can improve method generation when they are provided as context to LLMs. For this, we designed a prompting approach that iteratively provides mutation-selected unit tests to the model, requesting it to reconstruct the implementation of a given method based on the provided tests. We generated new implementations for 35 methods from two popular open-source JavaScript/TypeScript repositories, and evaluated their functional equivalence against their original implementations. Our results show that the pass rate improves as tests are added, with gains of up to 95 p.p. over the no-test baseline of the same method, showing that unit tests are a valuable source of information for LLMs when generating code.
Palavras-chave:
Large Language Models, Unit tests, Code generation, Mutation testing, Prompt engineering
Referências
Altino Alves, João Eduardo Montandon, and Andre Hora. 2026. Testing Framework Migration with Large Language Models. In 7th ACM/IEEE International Conference on Automation of Software Test (AST). 1–11.
Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. 1994. The Goal Question Metric Approach. Encyclopedia of Software Engineering (1994).
Kent Beck. 2003. Test Driven Development: By Example. Addison-Wesley Professional, Boston.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs] (July 2021).
Mike Cohn. 2010. Succeeding with Agile: Software Development Using Scrum. Addison-Wesley Professional, Upper Saddle River, NJ.
G. Comanici et al. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261 (2025).
date-fns. 2026. date-fns: Modern JavaScript Date Utility Library. [link]. Accessed June 2026.
Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (1978), 34–41.
Directus. 2026. Directus: The Modern Data Stack. [link]. Accessed June 2026.
Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation. IEEE Transactions on Software Engineering 50, 9 (2024), 2254–2268. DOI: 10.1109/TSE.2024.3428972
Michael Feathers. 2004. Working Effectively with Legacy Code (1st ed.). Pearson.
Mohammad Ghafari, Carlo Ghezzi, Andrea Mocci, and Giordano Tamburrelli. 2014. Mining unit tests for code recommendation. In Proceedings of the 22nd International Conference on Program Comprehension. 142–145.
Google. 2025. Gemini 2.5 Flash. [link]. Accessed June 2026.
Google. 2025. Gemini 3 Developer Guide. [link]. Accessed June 2026.
Google. 2025. Gemini API Pricing. [link]. Accessed June 2026.
Google. 2025. Thinking — Gemini API. [link]. Accessed June 2026.
Cristina Improta, Rosalia Tufano, Pietro Liguori, Domenico Cotroneo, and Gabriele Bavota. 2025. Quality In, Quality Out: Investigating Training Data’s Role in AI Code Generation. In 33rd International Conference on Program Comprehension (ICPC). 454–465.
Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. DOI: 10.1109/TSE.2010.62
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A Survey on Large Language Models for Code Generation. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 58:1–58:72.
Marcos Macedo, Yuan Tian, Pengyu Nie, Filipe R. Cogo, and Bram Adams. 2025. InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translation. In 47th International Conference on Software Engineering (ICSE). 1153–1164.
Noble Saji Mathews and Meiyappan Nagappan. 2024. Test-Driven Development and LLM-based Code Generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24). 1583–1594. DOI: 10.1145/3691620.3695527
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (2024).
David Sherret. 2026. ts-morph: TypeScript Compiler API Wrapper for Static Analysis and Programmatic Code Changes. [link]. Accessed June 2026.
Stryker Mutator. 2026. StrykerJS: Mutation Testing for JavaScript and TypeScript. [link]. Accessed June 2026.
Matt Wynne, Aslak Hellesoy, and Steve Tooke. 2017. The Cucumber Book: Behaviour-Driven Development for Testers and Developers. Pragmatic Bookshelf, Raleigh, North Carolina.
Zixiao Zhu, Yanzhen Zou, Bing Xie, Yong Jin, Zeqi Lin, and Lu Zhang. 2014. Mining API Usage Examples from Test Code. In 2014 IEEE International Conference on Software Maintenance and Evolution. 301–310.
Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. 1994. The Goal Question Metric Approach. Encyclopedia of Software Engineering (1994).
Kent Beck. 2003. Test Driven Development: By Example. Addison-Wesley Professional, Boston.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs] (July 2021).
Mike Cohn. 2010. Succeeding with Agile: Software Development Using Scrum. Addison-Wesley Professional, Upper Saddle River, NJ.
G. Comanici et al. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261 (2025).
date-fns. 2026. date-fns: Modern JavaScript Date Utility Library. [link]. Accessed June 2026.
Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (1978), 34–41.
Directus. 2026. Directus: The Modern Data Stack. [link]. Accessed June 2026.
Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation. IEEE Transactions on Software Engineering 50, 9 (2024), 2254–2268. DOI: 10.1109/TSE.2024.3428972
Michael Feathers. 2004. Working Effectively with Legacy Code (1st ed.). Pearson.
Mohammad Ghafari, Carlo Ghezzi, Andrea Mocci, and Giordano Tamburrelli. 2014. Mining unit tests for code recommendation. In Proceedings of the 22nd International Conference on Program Comprehension. 142–145.
Google. 2025. Gemini 2.5 Flash. [link]. Accessed June 2026.
Google. 2025. Gemini 3 Developer Guide. [link]. Accessed June 2026.
Google. 2025. Gemini API Pricing. [link]. Accessed June 2026.
Google. 2025. Thinking — Gemini API. [link]. Accessed June 2026.
Cristina Improta, Rosalia Tufano, Pietro Liguori, Domenico Cotroneo, and Gabriele Bavota. 2025. Quality In, Quality Out: Investigating Training Data’s Role in AI Code Generation. In 33rd International Conference on Program Comprehension (ICPC). 454–465.
Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. DOI: 10.1109/TSE.2010.62
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A Survey on Large Language Models for Code Generation. ACM Transactions on Software Engineering and Methodology 35, 2 (2026), 58:1–58:72.
Marcos Macedo, Yuan Tian, Pengyu Nie, Filipe R. Cogo, and Bram Adams. 2025. InterTrans: Leveraging Transitive Intermediate Translations to Enhance LLM-based Code Translation. In 47th International Conference on Software Engineering (ICSE). 1153–1164.
Noble Saji Mathews and Meiyappan Nagappan. 2024. Test-Driven Development and LLM-based Code Generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24). 1583–1594. DOI: 10.1145/3691620.3695527
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering 50, 1 (2024).
David Sherret. 2026. ts-morph: TypeScript Compiler API Wrapper for Static Analysis and Programmatic Code Changes. [link]. Accessed June 2026.
Stryker Mutator. 2026. StrykerJS: Mutation Testing for JavaScript and TypeScript. [link]. Accessed June 2026.
Matt Wynne, Aslak Hellesoy, and Steve Tooke. 2017. The Cucumber Book: Behaviour-Driven Development for Testers and Developers. Pragmatic Bookshelf, Raleigh, North Carolina.
Zixiao Zhu, Yanzhen Zou, Bing Xie, Yong Jin, Zeqi Lin, and Lu Zhang. 2014. Mining API Usage Examples from Test Code. In 2014 IEEE International Conference on Software Maintenance and Evolution. 301–310.
Publicado
11/09/2026
Como Citar
LIMA, Guilherme Mota Bromonschenkel; MONTANDON, João Eduardo.
Test-Driven Prompting: An Empirical Evaluation of Methods Reconstruction based on Unit Tests. In: WORKSHOP DE VISUALIZAÇÃO, EVOLUÇÃO E MANUTENÇÃO DE SOFTWARE (VEM), 14. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 41-47.
DOI: https://doi.org/10.5753/vem.2026.32508.
