A Framework for Characterizing What Makes an Instance Hard to Classify

• Maria Gabriela Valeriano ITA / UNIFESP https://orcid.org/0000-0002-3631-156X
• Pedro Yuri Arbs Paiva ITA https://orcid.org/0000-0001-9768-7563
• Carlos Roberto Veiga Kiffer UNIFESP https://orcid.org/0000-0003-1122-0693
• Ana Carolina Lorena ITA https://orcid.org/0000-0002-6140-571X

Resumo

The health domain has been largely benefited by Machine Learning solutions, which can be used for building predictive models to support medical decisions. But, for increasing the reliability of these systems, it is important to understand when the models are prone to failures. In this paper, we investigate what can we learn from the instances of a dataset which are hard to classify by Machine Learning models. Different reasons may explain why one or a set of instances are misclassified, despite the predictive model used. They can be either noisy, anomalous or placed in overlapping regions, to name a few. Our framework works at two levels: the original base dataset and a meta-dataset built to reflect the hardness level of the instances. A two-dimensional hardness embedding is assembled, which can be visually inspected to determine sets of instances to scrutinize better. We show some analysis that can be undertaken in this hardness space that allow to characterize why some of the instances are hard to classify, with case studies on health datasets.