Benchmarking Ranking Frameworks for Algorithmic Fairness

• Gabriel Garcia Almeida UNIFESP
• Lilian Berton UNIFESP

DOI: https://doi.org/10.5753/eniac.2026.31245

Resumo

Evaluating algorithmic fairness in machine learning pipelines presents a complex multi-criteria optimization challenge, often characterized by severe trade-offs between predictive utility and disparate impact mitigation. While numerous mathematical metrics and mitigation frameworks exist, systematically identifying the optimal configuration for a given operational context remains an open problem. This paper investigates the viability of ranking-based decision-support frameworks as a reliable paradigm for auditing and selecting fairness-aware machine learning models. We systematically evaluate three distinct multi-criteria aggregation paradigms, Multi-Criteria Performance Measure (MCPM), Multi-Objective Evaluation (MOE), and non-linear Borda-style ranking frameworks, across four benchmark datasets under varying sensitive attributes. Our empirical results reveal critical trade-offs between paradigm-specific stability and sensitivity, demonstrating that model selection is highly non-trivial and acutely sensitive to both the choice of evaluation framework and the baseline bias topology of the underlying data.

Referências

Agarwal, A., Beygelzimer, A., Dudik, M., Langford, J., and Wallach, H. (2018). A reductions approach to fair classification. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 60–69. PMLR.

Barocas, S., Hardt, M., and Narayanan, A. (2023). Fairness and Machine Learning: Limitations and Opportunities. MIT Press.

Barocas, S. and Selbst, A. D. (2016). Big data’s disparate impact. Calif. L. Rev., 104:671.

Caton, S. and Haas, C. (2024). Fairness in machine learning: A survey. ACM Computing Surveys, 56(7):1–38.

Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. (2002). Smote: synthetic minority over-sampling technique. J. Artif. Int. Res., 16(1):321–357.

Chia, P. J., Attanasio, G., Tagliabue, J., Bianchi, F., Greco, C., Moreira, G. d. S. P., Eynard, D., and Husain, F. (2022). E pluribus unum: Guidelines on multi-objective evaluation of recommender systems. Journal of the ACM (JACM), 37(4):1–15.

dos Santos Porphirio, J., dos Santos, D. J., Azevedo, S., and Berton, L. (2025). Analyzing data bias through data augmentation: A case study in financial data. In Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), pages 950–961. SBC.

Fatoretto, M. B., Özbulak, G., Anjos, A., and Berton, L. (2025). Optimizing fairness and utility in healthcare machine learning models. Applied Soft Computing, 181:113426.

Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. (2014). Certifying and removing disparate impact.

Goyal, S. (2020). Credit Card customers. Kaggle. URL: [link].

Hardt, M., Price, E., and Srebro, N. (2016). Equality of opportunity in supervised learning. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.

Hofmann, H. (1994). Statlog (German Credit Data). UCI Machine Learning Repository. DOI: 10.24432/C5NC77.

Kamiran, F. and Calders, T. (2011). Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems, 33(1):1–33.

Kamiran, F., Karim, A., and Zhang, X. (2012). Decision theory for freedom of choice in fair classification: Reject option-based approach. In 2012 IEEE 12th International Conference on Data Mining (ICDM), pages 378–387. IEEE.

Kim, E. and UCI Machine Learning (2005). Default of Credit Card Clients Dataset. Kaggle. URL: [link].

Kusner, M. J., Loftus, J. R., Russell, C., and Silva, R. (2017). Counterfactual fairness. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems 30 (NIPS 2017), pages 4066–4076.

Leeuw, J. d., Hornik, K., and Mair, P. (2009). Isotone optimization in r: Pool-adjacentviolators algorithm (pava) and active set methods. Journal of Statistical Software, 32(5).

MathWorks (2024). Disparate impact remover. [link]. Acess in october 22th 2025.

Minatel, D. (2024). O Funcionamento Diferencial dos Itens na Indução de Classificadores Mais Justos. Tese (doutorado em ciências — ciências de computação e matemática computacional), Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, São Carlos.

MoneyMan (2019). Credit Card Approval Prediction. Kaggle. URL: [link].

Pérez, P. (2025). Enhancing Fairness in Machine Learning: Identifying and Mitigating Bias with a Focus on Gender Bias in Finance. PhD thesis, Open University.

Rabonato, R. T. and Berton, L. (2025). A systematic review of fairness in machine learning. AI and Ethics, 5(3):1943–1954.

Silva, B. P. M. and Berton, L. (2025). Analyzing the trade-off between fairness and model performance in supervised learning: A case study in the mimic dataset. In Simpósio Brasileiro de Computação Aplicada à Saúde (SBCAS), pages 212–223. SBC.

Trinh, T. K. and Zhang, D. (2024). Algorithmic fairness in financial decision-making: Detection and mitigation of bias in credit scoring applications. Journal of Advanced Computing Systems (JACS), 4(2):36–49.