Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews

• Gilberto Sussumu Hida CESAR School
• Danilo Monteiro Ribeiro CESAR School
• Clayton Suguio Hida UEAP

DOI: https://doi.org/10.5753/eniac.2026.29779

Resumo

This study analyses LLMs in imbalanced binary classification, using study screening in systematic reviews as the application domain. An experiment was conducted in five reviews, comparing individual and batch processing, with and without prevalence metadata. The results indicate a limited influence of the prevalence metadata, with no evidence that it improves performance. In contrast, batch processing produced larger behavioral changes that varied according to the prevalence of the class. The aggregate and item-level analyses did not always coincide. Therefore, batch processing should be evaluated not only in terms of cost, but also in relation to its effects on decision-making behavior.

Referências

Bestgen, Y. (2022). Please, don’t forget the difference and the confidence interval when seeking for the state-of-the-art status. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5956–5962.

Bornmann, L., Haunschild, R., and Mutz, R. (2021). Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1):224.

Chai, K. E., Lines, R. L., Gucciardi, D. F., and Ng, L. (2021). Research screener: a machine learning tool to semi-automate abstract screening for systematic reviews. Systematic reviews, 10(1):93.

Cheng, Z., Kasai, J., and Yu, T. (2023). Batch prompting: Efficient inference with large language model apis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 792–810.

Fagerberg, P., Sallander, O., Patil, K. V., Berg, A., Nyman, A., Borg, N., and Lindén, T. (2026). Batch size effects on mid-2025 state-of-the-art large language model performance in automated title and abstract screening. Cochrane Evidence Synthesis and Methods, 4(3):e70082.

Hida, G. S. and Do Nascimento, A. C. A. (2025). Overview of machine learning in class imbalance scenarios: Trends, challenges, and approaches. Expert Systems with Applications, page 129592.

Hida, G. S., Ribeiro, D. M., and Hida, C. S. (2026a). Research Artifacts: Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews. Zenodo record. DOI: 10.5281/zenodo.20414963.

Hida, G. S., Ribeiro, D. M., and Yahata, E. (2026b). Beyond accuracy: Llm variability in evidence screening for software engineering slrs. arXiv preprint arXiv:2604.27006.

Huotala, A., Kuutila, M., and Mäntylä, M. (2025). Sesr-eval: Dataset for evaluating llms in the title-abstract screening of systematic reviews. In 2025 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 01–12. IEEE.

Huotala, A., Kuutila, M., Ralph, P., and Mäntylä, M. (2024). The promise and challenges of using llms to accelerate the screening process of systematic reviews. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, pages 262–271.

Kitchenham, B. et al. (2004). Procedures for performing systematic reviews. Keele, UK, Keele University, 33(2004):1–26.

Kuutila, M., Mäntylä, M., Farooq, U., and Claes, M. (2020). Time pressure in software engineering: A systematic review. Information and Software Technology, 121:106257.

Laban, P., Murakhovs’ ka, L., Xiong, C., and Wu, C.-S. (2023). Are you sure? challenging llms leads to performance drops in the flipflop experiment. arXiv preprint arXiv:2311.08596.

Lewowski, T. and Madeyski, L. (2022). How far are we from reproducible research on code smell detection? a systematic literature review. Information and Software Technology, 144:106783.

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12:157–173.

Madeyski, L., Kitchenham, B., and Shepperd, M. (2025). Llm4screenlit: Recommendations on assessing the performance of large language models for screening literature in systematic reviews. arXiv preprint arXiv:2511.12635.

Meta AI (2024). Llama 3.3 70B Instruct. Model card. Hugging Face. Released on December 6, 2024. Accessed May 2026.

OpenAI (2025). GPT-5 mini (gpt-5-mini-2025-08-07). OpenAI API documentation. Snapshot: gpt-5-mini-2025-08-07. Accessed May 2026.

Pipal, C., Vogel, E.-M., Wack, M., and Esser, F. (2026). Researchers waste 80% of llm annotation costs by classifying one text at a time. arXiv preprint arXiv:2604.03684.

Rani, P., Blasi, A., Stulova, N., Panichella, S., Gorla, A., and Nierstrasz, O. (2023). A decade of code comment quality assessment: A systematic literature review. Journal of Systems and Software, 195:111515.

Sharbaf, M., Zamani, B., and Sunyé, G. (2023). Conflict management techniques for model merging: a systematic mapping review. Software and Systems Modeling, 22(3):1031–1079.

Syriani, E., David, I., and Kumar, G. (2024). Screening articles for systematic reviews with chatgpt. Journal of Computer Languages, 80:101287.

Tambon, F., Laberge, G., An, L., Nikanjam, A., Mindom, P. S. N., Pequignot, Y., Khomh, F., Antoniol, G., Merlo, E., and Laviolette, F. (2022). How to certify machine learning based safety-critical systems? a systematic literature review. Automated Software Engineering, 29(2):38.

Together AI (2024). Together AI inference api. API documentation. Accessed May 2026.

Vach, W. and Gerke, O. (2023). Gwet’s ac1 is not a substitute for cohen’s kappa–a comparison of basic properties. MethodsX, 10:102212.

Wang, S., Scells, H., Zhuang, S., Potthast, M., Koopman, B., and Zuccon, G. (2024). Zero-shot generative large language models for systematic review screening automation. In European Conference on Information Retrieval, pages 403–420. Springer.