Toward a Fair Evaluation Protocol for Automatic Prompt Optimization Methods
Resumo
Large Language Models are highly sensitive to prompt phrasing, motivating many Automatic Prompt Optimization (APO) methods that search the prompt space without gradients. Comparing them fairly is hard because papers report results under different budgets, datasets, seeds, and base models. We introduce APOBENCH, a five-pillar evaluation protocol standardizing budget allocation, seed control, benchmark diversity, multi-metric reporting, and model parity. Validating it on seven gradient-free methods across BBH, GSM8K, and HumanEval, we find that no method dominates: the lightweight ones use up to 11× fewer tokens than the most accurate while losing little accuracy, and the stop reason exposes convergence differences that accuracy alone hides.
Palavras-chave:
Large Language Models, Prompt Optimization, Automated Machine Learning
Referências
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877–1901.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
Cui, W., Zhang, J., Li, Z., Sun, H., Lopez, D., Das, K., Malin, B. A., and Kumar, S. (2025). SEE: Strategic exploration and exploitation for cohesive in-context prompt optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), Volume 1: Long Papers, pages 29575–29627, Vienna, Austria.
Fernando, C., Banarse, D., Michalewski, H., Osindero, S., and Rocktäschel, T. (2024). Promptbreeder: Self-referential self-improvement via prompt evolution. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235, pages 13481–13544.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. (2024). A framework for few-shot language model evaluation.
Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. (2024). Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024).
He, H., Liu, Q., Xu, L., Shivade, C., Zhang, Y., Srinivasan, S., and Kirchhoff, K. (2025). CriSPO: Multi-aspect critique-suggestion-guided automatic prompt optimization for text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24014–24022.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. (2024). DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024).
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP ’23).
Li, W., Wang, X., Li, W., and Jin, B. (2025). A survey of automatic prompt engineering: An optimization perspective. arXiv preprint arXiv:2502.11560.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., Wang, J., Santhanam, K., Orr, L., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N., Khattab, O., Henderson, P., Huang, Q., Chi, R., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research.
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., and Stanovsky, G. (2024). State of what art? A call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949.
Pan, R., Xing, S., Diao, S., Sun, W., Liu, X., Shum, K., Zhang, J., Pi, R., and Zhang, T. (2024). PLUM: Prompt learning using metaheuristics. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2177–2197.
Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., and Zeng, M. (2023). Automatic prompt optimization with “Gradient Descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 7957–7968, Singapore.
Ramnath, K. et al. (2025). A systematic survey of automatic prompt optimization techniques. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), Main, pages 33078–33110, Suzhou, China.
Saletta, M. and Ferretti, C. (2024). Exploring the prompt space of large language models through evolutionary sampling. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’24), pages 1345–1353.
Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. (2024). Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024).
Sécheresse, X., Guilbert-Ly, J.-Y., and Villedieu de Torcy, A. (2025). GAAPO: Genetic algorithmic applied to prompt optimization. Frontiers in Artificial Intelligence, 8. Also presented at ECAI 2025.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., and Wei, J. (2023). Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. (2022). Emergent abilities of large language models. Transactions on Machine Learning Research.
Wen, B., Ke, P., Sun, Y., Wang, C., Gu, X., Zhou, J., Tang, J., Wang, H., and Huang, M. (2025). HPSS: Heuristic prompting strategy search for LLM evaluators. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24974–25007, Vienna, Austria.
Yang, A. et al. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. (2024). Large language models as optimizers. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024).
Zehle, T., Heiß, T., Schlager, M., Aßenmacher, M., and Feurer, M. (2026). promptolution: A unified, modular framework for prompt optimization. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 282–296, Rabat, Morocco.
Zehle, T., Schlager, M., Heiß, T., and Feurer, M. (2025). CAPO: Cost-aware prompt optimization. In Proceedings of the Fourth International Conference on Automated Machine Learning, volume 293 of Proceedings of Machine Learning Research.
Zheng, W., Das, S. S. S., Zhang, Y., and Zhang, R. (2025). GreaterPrompt: A unified, customizable, and high-performing open-source toolkit for prompt optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025): System Demonstrations.
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. (2023). Large language models are human-level prompt engineers. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR 2023).
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Gong, N. Z., Zhang, Y., and Xie, X. (2023). PromptBench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528.
Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., and Chen, K. (2024). ProSA: Assessing and understanding the prompt sensitivity of LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950–1976, Miami, Florida, USA.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
Cui, W., Zhang, J., Li, Z., Sun, H., Lopez, D., Das, K., Malin, B. A., and Kumar, S. (2025). SEE: Strategic exploration and exploitation for cohesive in-context prompt optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), Volume 1: Long Papers, pages 29575–29627, Vienna, Austria.
Fernando, C., Banarse, D., Michalewski, H., Osindero, S., and Rocktäschel, T. (2024). Promptbreeder: Self-referential self-improvement via prompt evolution. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235, pages 13481–13544.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. (2024). A framework for few-shot language model evaluation.
Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. (2024). Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024).
He, H., Liu, Q., Xu, L., Shivade, C., Zhang, Y., Srinivasan, S., and Kirchhoff, K. (2025). CriSPO: Multi-aspect critique-suggestion-guided automatic prompt optimization for text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24014–24022.
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. (2024). DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024).
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP ’23).
Li, W., Wang, X., Li, W., and Jin, B. (2025). A survey of automatic prompt engineering: An optimization perspective. arXiv preprint arXiv:2502.11560.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., Wang, J., Santhanam, K., Orr, L., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N., Khattab, O., Henderson, P., Huang, Q., Chi, R., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research.
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., and Stanovsky, G. (2024). State of what art? A call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949.
Pan, R., Xing, S., Diao, S., Sun, W., Liu, X., Shum, K., Zhang, J., Pi, R., and Zhang, T. (2024). PLUM: Prompt learning using metaheuristics. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2177–2197.
Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., and Zeng, M. (2023). Automatic prompt optimization with “Gradient Descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 7957–7968, Singapore.
Ramnath, K. et al. (2025). A systematic survey of automatic prompt optimization techniques. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), Main, pages 33078–33110, Suzhou, China.
Saletta, M. and Ferretti, C. (2024). Exploring the prompt space of large language models through evolutionary sampling. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’24), pages 1345–1353.
Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. (2024). Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024).
Sécheresse, X., Guilbert-Ly, J.-Y., and Villedieu de Torcy, A. (2025). GAAPO: Genetic algorithmic applied to prompt optimization. Frontiers in Artificial Intelligence, 8. Also presented at ECAI 2025.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., and Wei, J. (2023). Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003–13051, Toronto, Canada.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. (2022). Emergent abilities of large language models. Transactions on Machine Learning Research.
Wen, B., Ke, P., Sun, Y., Wang, C., Gu, X., Zhou, J., Tang, J., Wang, H., and Huang, M. (2025). HPSS: Heuristic prompting strategy search for LLM evaluators. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24974–25007, Vienna, Austria.
Yang, A. et al. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. (2024). Large language models as optimizers. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024).
Zehle, T., Heiß, T., Schlager, M., Aßenmacher, M., and Feurer, M. (2026). promptolution: A unified, modular framework for prompt optimization. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 282–296, Rabat, Morocco.
Zehle, T., Schlager, M., Heiß, T., and Feurer, M. (2025). CAPO: Cost-aware prompt optimization. In Proceedings of the Fourth International Conference on Automated Machine Learning, volume 293 of Proceedings of Machine Learning Research.
Zheng, W., Das, S. S. S., Zhang, Y., and Zhang, R. (2025). GreaterPrompt: A unified, customizable, and high-performing open-source toolkit for prompt optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025): System Demonstrations.
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. (2023). Large language models are human-level prompt engineers. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR 2023).
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Gong, N. Z., Zhang, Y., and Xie, X. (2023). PromptBench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528.
Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., and Chen, K. (2024). ProSA: Assessing and understanding the prompt sensitivity of LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950–1976, Miami, Florida, USA.
Publicado
19/10/2026
Como Citar
SILVA, Raul José Silvério da; POZO, Aurora Trinidad Ramirez.
Toward a Fair Evaluation Protocol for Automatic Prompt Optimization Methods. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 23. , 2026, Cuiabá/MT.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 1028-1039.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2026.30795.
