The Impact of Model Selection Metrics during Hyperparameter Tuning on Algorithmic Fairness: An Empirical Study

• Bianca Matos de Barros UFRGS https://orcid.org/0000-0003-0818-1267
• Diego Dimer Rodrigues UFRGS https://orcid.org/0000-0002-9544-117X
• Gabriela Bellardinelli Oliveira UFRGS
• Mariana Recamonde-Mendoza UFRGS / HCPA https://orcid.org/0000-0003-2800-1032

DOI: https://doi.org/10.5753/semish.2026.23608

Resumo

The growing use of machine learning in high-stakes domains raises concerns about fairness. The role of optimization metrics in shaping these outcomes remains underexplored. Using a controlled setup, this study investigates how seven performance metrics used for hyperparameter tuning and model selection affect fairness outcomes across five benchmark datasets. Results show that metrics are not neutral: recall-based optimization yields higher disparities, while precision and specificity lead to more balanced outcomes, with PR-AUC showing intermediate behavior. Overall, metric choice influences fairness, but outcomes are largely driven by dataset characteristics, with optimization redistributing errors rather than eliminating bias.

Referências

Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., and Wallach, H. (2018). A reductions approach to fair classification. In International Conference on Machine Learning (ICML), pages 60–69. PMLR.

Almasi, M., Nezami, N., Di Carlo, F., Asudeh, A., and Anahideh, H. (2026). Adaptive pareto exploration (apex) for fairness-aware hyperparameter optimization in fairpilot. Information and Software Technology, 189(C).

Broussard, M. (2018). Artificial unintelligence: How computers misunderstand the world. mit Press.

Buolamwini, J. and Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency (FAccT), pages 77–91. PMLR.

Castelnovo, A., Crupi, R., Greco, G., Regoli, D., Penco, I. G., and Cosentini, A. C. (2022). A clarification of the nuances in the fairness metrics landscape. Scientific reports, 12(1):4209.

Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2):153–163.

Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012). Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214–226.

Gaudreault, J.-G., Branco, P., and Gama, J. (2021). An analysis of performance metrics for imbalanced classification. In International Conference on Discovery Science, pages 67–77. Springer.

Hardt, M., Price, E., and Srebro, N. (2016). Equality of opportunity in supervised learning. In Advances in neural information processing systems (NIPS), pages 3315–3323.

Islam, R., Pan, S., and Foulds, J. R. (2021). Can we obtain fairness for free? In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 586–596.

Kamiran, F. and Calders, T. (2012). Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems, 33(1):1–33.

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., and Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM Computing Surveys, 54(6):1–35.

Nadeem, M. A. (2025). Optimizing fairness in machine learning: A hyperparameter tuning approach. In 2025 IEEE International Conference on Omni-layer Intelligent Systems (COINS), pages 1–5. IEEE.

Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366:447–453.

O’Neil, C. (2017). Weapons of math destruction: How big data increases inequality and threatens democracy. Crown.

Rabonato, R. T. and Berton, L. (2025). A systematic review of fairness in machine learning. AI and Ethics, 5(3):1943–1954.

Suresh, H. and Guttag, J. (2021). A framework for understanding sources of harm throughout the machine learning life cycle. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–9.