Assessing Fair Machine Learning Strategies Through a Fairness-Utility Trade-off Metric

• Luiz Fernando F. P. de Lima UFPB
• Danielle Rousy D. Ricarte UFPB
• Clauirton A. Siebra UFPB

DOI: https://doi.org/10.5753/eniac.2021.18288

Resumo

Due to the increasing use of artificial intelligence for decision making and the observation of biased decisions in many applications, researchers are investigating solutions that attempt to build fairer models that do not reproduce discrimination. Some of the explored strategies are based on adversarial learning to achieve fairness in machine learning by encoding fairness constraints through an adversarial model. Moreover, it is usual for each proposal to assess its model with a specific metric, making comparing current approaches a complex task. In that sense, we defined a utility and fairness trade-off metric. We assessed 15 fair model implementations and a baseline model using this metric, providing a systemically comparative ruler for other approaches.

Referências

ANGWIN, J., LARSON, J., MATTU, S., and KIRCHNER, L. (2016). Machine bias: There’s software used across the country to predict future criminals. And it’s biased against blacks. ProPublica. [link].

BEUTEL, A., CHEN, J., ZHAO, Z., and CHI, E. H. (2017). Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv:1707.00075.

BOLUKBASI, T., CHANG, K.-W., ZOU, J. Y., SALIGRAMA, V., and KALAI, A. T. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems, pages 4349–4357.

BOUSMALIS, K., TRIGEORGIS, G., SILBERMAN, N., KRISHNAN, D., and ERHAN, D. (2016). Domain separation networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 343–351.

CALDERS, T., KAMIRAN, F., and PECHENIZKIY, M. (2009). Building classifiers with independency constraints. In 2009 IEEE International Conference on Data Mining Workshops, pages 13–18. IEEE.

DWORK, C., HARDT, M., PITASSI, T., REINGOLD, O., and ZEMEL, R. (2012). Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214–226.

GANIN, Y., USTINOVA, E., AJAKAN, H., GERMAIN, P., LAROCHELLE, H., LAVIOLETTE, F., MARCHAND, M., and LEMPITSKY, V. (2016). Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096– 2030.

GARCIA, M. (2016). Racist in the machine: The disturbing implications of algorithmic bias. World Policy Journal, 33(4):111–117.

HAO, K. (2020). The uk exam debacle reminds us that algorithms can’t fix broken systems. MIT Technology Review. [link].

HARDT, M., PRICE, E., and SREBRO, N. (2016). Equality of opportunity in supervised learning. In Advances in neural information processing systems, pages 3315–3323.

JONES, G. P., HICKEY, J. M., DI STEFANO, P. G., DHANJAL, C., STODDART, L. C., and VASILEIOU, V. (2020). Metrics and methods for a systematic comparison of fairness-aware machine learning algorithms. arXiv preprint arXiv:2010.03986.

KEARNS, M. and ROTH, A. (2019). The ethical algorithm: The science of socially aware algorithm design. Oxford University Press.

LEAVY, S. (2018). Gender bias in artificial intelligence: The need for diversity and gender theory in machine learning. In Proceedings of the 1st international workshop on gender equality in software engineering, pages 14–16.

LUM, K. and JOHNDROW, J. (2016). A statistical framework for fair predictive algorithms. arXiv preprint arXiv:1610.08077.

MADRAS, D., CREAGER, E., PITASSI, T., and ZEMEL, R. (2018). Learning adversarially fair and transferable representations. In Proceedings of the 35th International Conference on Machine Learning, pages 3384–3393.

MEHRABI, N., MORSTATTER, F., SAXENA, N., LERMAN, K., and GALSTYAN, A. (2019). A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635.

VERMA, S. and RUBIN, J. (2018). Fairness definitions explained. In 2018 IEEE/ACM International Workshop on Software Fairness (FairWare), pages 1–7. IEEE.

WAZLAWICK, R. S. (2020). Metodologia de pesquisa para ciência da computação. GEN LTC, 3 edition.

ZHANG, B. H., LEMOINE, B., and MITCHELL, M. (2018). Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340.