Evaluating the Effectiveness of Visual Representations of SHAP Values Toward Explainable Artificial Intelligence
Abstract
The employment of Machine Learning (ML) and Deep Learning models across diverse domains has grown exponentially in recent years. These models undertake critical tasks spanning medical diagnoses, criminal sentencing, and loan approvals. Consequently, there is a need for these models to be interpretable, enabling users to grasp the rationale behind predictions and engendering trust. Equally vital is the capability of developers to pinpoint and rectify any erroneous behaviors. In this context emerges the field of Explainable AI (XAI), which aims to develop methods to make ML models more interpretable while maintaining their performance level. Various methods have been proposed, many leveraging visual explanations to elucidate model behavior. However, a notable gap remains: a lack of rigorous assessment regarding the effectiveness of these explanations in enhancing interpretability. In this paper, we evaluate whether the SHAP method, widely used in the XAI field, achieves its objective of making a model more interpretable. We conducted a study based on the concept of simulatability. We asked participants who have mathematical or statistical backgrounds and intermediate to advanced knowledge of machine learning and model interpretability to classify some given instances of a dataset, presented first without the explanations provided by the SHAP method and then with them. The goal was to assess whether the explanation made the participants better at classifying the instances. Our findings reveal that the visualizations can be confusing even for users with a mathematical background. Additionally, we argue that there is a need for XAI researchers to work collaboratively with visual analytics experts to develop these visualizations, as well as test the visualizations with users of various backgrounds.
Keywords:
Visualization, Explainable Artificial Intelligence, SHAP values
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity Checks for Saliency Maps. In Advances in Neural Information Processing Systems, Vol. 31. Curran Associates, Inc. [link]
Sajid Ali, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M. AlonsoMoral, Roberto Confalonieri, Riccardo Guidotti, Javier Del Ser, Natalia DíazRodríguez, and Francisco Herrera. 2023. Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence. Information Fusion 99 (Nov. 2023), 101805. DOI: 10.1016/j.inffus.2023.101805.
Gulsum Alicioglu and Bo Sun. 2022. A survey of visual analytics for Explainable Artificial Intelligence methods. Computers & Graphics 102 (Feb. 2022), 502–520. DOI: 10.1016/j.cag.2021.09.002
Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 8–13.
A. Chatzimparmpas, R. M. Martins, I. Jusufi, K. Kucher, F. Rossi, and A. Kerren. 2020. The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations. Computer Graphics Forum 39, 3 (2020), 713–756. DOI: 10.1111/cgf.14034 _eprint: [link].
Michael Correll. 2019. Ethical Dimensions of Visualization Research. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13. DOI: 10.1145/3290605.3300418 arXiv:1811.07271 [cs].
Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. [link] arXiv:1702.08608 [cs, stat].
David Gunning and David W. Aha. 2019. DARPA’s Explainable Artificial Intelligence Program. AI Magazine 40, 2 (2019), 44–58. DOI: 10.1609/aimag.v40i2.2850 arXiv: [link]
A. Hart. 2001. Mann-Whitney test is not just a test of medians: differences in spread can be important. BMJ 323, 7309 (Aug. 2001), 391–393. DOI: 10.1136/bmj.323.7309.391
Peter Hase and Mohit Bansal. 2020. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? [link] arXiv:2005.01831 [cs].
Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018. Metrics for Explainable AI: Challenges and Prospects. (2018).
Been Kim. 2015. Interactive and Interpretable Machine Learning Models for Human Machine Collaboration. (2015).
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). [link] arXiv:1711.11279 [stat].
Himabindu Lakkaraju, Stephen H. Bach, and Jure Leskovec. 2016. Interpretable Decision Sets: A Joint Framework for Description and Prediction. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, San Francisco California USA, 1675–1684. DOI: 10.1145/2939672.2939874
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. 2019. Faithful and Customizable Explanations of Black Box Models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. ACM, Honolulu HI USA, 131–138. DOI: 10.1145/3306618.3314229
Yiran Li, Takanori Fujiwara, Yong K. Choi, Katherine K. Kim, and Kwan-Liu Ma. 2020. A visual analytics system for multi-model comparison on clinical data predictions. Visual Informatics 4, 2 (June 2020), 122–131. DOI: 10.1016/ j.visinf.2020.04.005
Zachary C. Lipton. 2017. The Mythos of Model Interpretability. [link] arXiv:1606.03490 [cs, stat].
Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. 2013. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, Chicago Illinois USA, 623–631. DOI: 10.1145/2487575.2487579
Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc. [link]
Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4765–4774. [link]
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (Feb. 2019), 1–38. DOI: 10.1016/j. artint.2018.07.007
Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019. Explaining Explanations in AI. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19). Association for Computing Machinery, New York, NY, USA, 279–288. DOI: 10.1145/3287560.3287574
Brett Poulin, Roman Eisner, Duane Szafron, Paul Lu, Russ Greiner, D S Wishart, Alona Fyshe, Brandon Pearcy, Cam MacDonell, and John Anvik. 2006. Visual Explanation of Evidence in Additive Classifiers. (2006).
Oliver Radley-Gardner, Hugh Beale, and Reinhard Zimmermann (Eds.). 2016. Fundamental Texts On European Private Law. Hart Publishing. DOI: 10.5040/9781782258674
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. [link] arXiv:1602.04938 [cs, stat].
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: HighPrecision Model-Agnostic Explanations. Proceedings of the AAAI Conference on Artificial Intelligence 32, 1 (April 2018). DOI: 10.1609/aaai.v32i1.11491
Greg Ridgeway, David Madigan, Thomas Richardson, and John O’Kane. 1998. Interpretable Boosted Naïve Bayes Classification. (1998).
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. [link] arXiv:1911.02508 [cs, stat].
J W Smith, J E Everhart, W C Dickson, W C Knowler, and R S Johannes. 1988. Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus. Proc Annu Symp Comput Appl Med Care (1988), 261–265.
Thilo Spinner, Udo Schlegel, Hanna Schafer, and Mennatallah El-Assady. 2019. explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning. IEEE Transactions on Visualization and Computer Graphics (2019), 1–1. DOI: 10.1109/TVCG.2019.2934629
Paolo Tamagnini, Josua Krause, Aritra Dasgupta, and Enrico Bertini. 2017. Interpreting Black-Box Classifiers Using Instance-Level Visual Explanations. In Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. ACM, Chicago IL USA, 1–6. DOI: 10.1145/3077257.3077260
Claire Woodcock, Brent Mittelstadt, Dan Busbridge, and Grant Blank. 2021. The Impact of Explanations on Layperson Trust in Artificial Intelligence–Driven Symptom Checker Apps: Experimental Study. Journal of Medical Internet Research 23, 11 (Nov. 2021), e29386. DOI: 10.2196/29386
Sajid Ali, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M. AlonsoMoral, Roberto Confalonieri, Riccardo Guidotti, Javier Del Ser, Natalia DíazRodríguez, and Francisco Herrera. 2023. Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence. Information Fusion 99 (Nov. 2023), 101805. DOI: 10.1016/j.inffus.2023.101805.
Gulsum Alicioglu and Bo Sun. 2022. A survey of visual analytics for Explainable Artificial Intelligence methods. Computers & Graphics 102 (Feb. 2022), 502–520. DOI: 10.1016/j.cag.2021.09.002
Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 workshop on explainable AI (XAI), Vol. 8. 8–13.
A. Chatzimparmpas, R. M. Martins, I. Jusufi, K. Kucher, F. Rossi, and A. Kerren. 2020. The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations. Computer Graphics Forum 39, 3 (2020), 713–756. DOI: 10.1111/cgf.14034 _eprint: [link].
Michael Correll. 2019. Ethical Dimensions of Visualization Research. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13. DOI: 10.1145/3290605.3300418 arXiv:1811.07271 [cs].
Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning. [link] arXiv:1702.08608 [cs, stat].
David Gunning and David W. Aha. 2019. DARPA’s Explainable Artificial Intelligence Program. AI Magazine 40, 2 (2019), 44–58. DOI: 10.1609/aimag.v40i2.2850 arXiv: [link]
A. Hart. 2001. Mann-Whitney test is not just a test of medians: differences in spread can be important. BMJ 323, 7309 (Aug. 2001), 391–393. DOI: 10.1136/bmj.323.7309.391
Peter Hase and Mohit Bansal. 2020. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? [link] arXiv:2005.01831 [cs].
Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018. Metrics for Explainable AI: Challenges and Prospects. (2018).
Been Kim. 2015. Interactive and Interpretable Machine Learning Models for Human Machine Collaboration. (2015).
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). [link] arXiv:1711.11279 [stat].
Himabindu Lakkaraju, Stephen H. Bach, and Jure Leskovec. 2016. Interpretable Decision Sets: A Joint Framework for Description and Prediction. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, San Francisco California USA, 1675–1684. DOI: 10.1145/2939672.2939874
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. 2019. Faithful and Customizable Explanations of Black Box Models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. ACM, Honolulu HI USA, 131–138. DOI: 10.1145/3306618.3314229
Yiran Li, Takanori Fujiwara, Yong K. Choi, Katherine K. Kim, and Kwan-Liu Ma. 2020. A visual analytics system for multi-model comparison on clinical data predictions. Visual Informatics 4, 2 (June 2020), 122–131. DOI: 10.1016/ j.visinf.2020.04.005
Zachary C. Lipton. 2017. The Mythos of Model Interpretability. [link] arXiv:1606.03490 [cs, stat].
Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. 2013. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, Chicago Illinois USA, 623–631. DOI: 10.1145/2487575.2487579
Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc. [link]
Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4765–4774. [link]
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 267 (Feb. 2019), 1–38. DOI: 10.1016/j. artint.2018.07.007
Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019. Explaining Explanations in AI. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19). Association for Computing Machinery, New York, NY, USA, 279–288. DOI: 10.1145/3287560.3287574
Brett Poulin, Roman Eisner, Duane Szafron, Paul Lu, Russ Greiner, D S Wishart, Alona Fyshe, Brandon Pearcy, Cam MacDonell, and John Anvik. 2006. Visual Explanation of Evidence in Additive Classifiers. (2006).
Oliver Radley-Gardner, Hugh Beale, and Reinhard Zimmermann (Eds.). 2016. Fundamental Texts On European Private Law. Hart Publishing. DOI: 10.5040/9781782258674
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. [link] arXiv:1602.04938 [cs, stat].
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: HighPrecision Model-Agnostic Explanations. Proceedings of the AAAI Conference on Artificial Intelligence 32, 1 (April 2018). DOI: 10.1609/aaai.v32i1.11491
Greg Ridgeway, David Madigan, Thomas Richardson, and John O’Kane. 1998. Interpretable Boosted Naïve Bayes Classification. (1998).
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. [link] arXiv:1911.02508 [cs, stat].
J W Smith, J E Everhart, W C Dickson, W C Knowler, and R S Johannes. 1988. Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus. Proc Annu Symp Comput Appl Med Care (1988), 261–265.
Thilo Spinner, Udo Schlegel, Hanna Schafer, and Mennatallah El-Assady. 2019. explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning. IEEE Transactions on Visualization and Computer Graphics (2019), 1–1. DOI: 10.1109/TVCG.2019.2934629
Paolo Tamagnini, Josua Krause, Aritra Dasgupta, and Enrico Bertini. 2017. Interpreting Black-Box Classifiers Using Instance-Level Visual Explanations. In Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. ACM, Chicago IL USA, 1–6. DOI: 10.1145/3077257.3077260
Claire Woodcock, Brent Mittelstadt, Dan Busbridge, and Grant Blank. 2021. The Impact of Explanations on Layperson Trust in Artificial Intelligence–Driven Symptom Checker Apps: Experimental Study. Journal of Medical Internet Research 23, 11 (Nov. 2021), e29386. DOI: 10.2196/29386
Published
2024-11-07
How to Cite
MOREIRA CUNHA, Bianca; DINIZ JUNQUEIRA BARBOSA, Simone.
Evaluating the Effectiveness of Visual Representations of SHAP Values Toward Explainable Artificial Intelligence. In: BRAZILIAN SYMPOSIUM ON HUMAN FACTORS IN COMPUTATIONAL SYSTEMS (IHC), 23. , 2024, Brasília/DF.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 622-632.
