Skip to main content
SpringerLink
Log in

A Framework for Characterizing What Makes an Instance Hard to Classify

  • Conference paper
  • First Online:
Intelligent Systems (BRACIS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14196))

Included in the following conference series:

  • 261 Accesses

Abstract

The health domain has been largely benefited by Machine Learning solutions, which can be used for building predictive models to support medical decisions. But, for increasing the reliability of these systems, it is important to understand when the models are prone to failures. In this paper, we investigate what can we learn from the instances of a dataset which are hard to classify by Machine Learning models. Different reasons may explain why one or a set of instances are misclassified, despite the predictive model used. They can be either noisy, anomalous or placed in overlapping regions, to name a few. Our framework works at two levels: the original base dataset and a meta-dataset built to reflect the hardness level of the instances. A two-dimensional hardness embedding is assembled, which can be visually inspected to determine sets of instances to scrutinize better. We show some analysis that can be undertaken in this hardness space that allow to characterize why some of the instances are hard to classify, with case studies on health datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    (https://github.com/gabivaleriano/explaining_healthdata).

References

  1. Anderson, D., Bjarnadottir, M.V., Nenova, Z.: Machine learning in healthcare: operational and financial impact. In: Babich, V., Birge, J.R., Hilary, G. (eds.) Innovative Technology at the Interface of Finance and Operations, vol. 11, pp. 153–174. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-75729-8_5

  2. Imrie, F., Cebere, B., McKinney, E.F., van der Schaar M.: AutoPrognosis 2.0: democratizing diagnostic and prognostic modeling in healthcare with automated machine learning. arXiv preprint arXiv:2210.12090 (2022)

  3. de Moraes, B.A.F., Miraglia, J., Donato, T., Filho, A.: Covid-19 diagnosis prediction in emergency care patients: a machine learning approach. MedRxiv, 2020-04 (2020)

    Google Scholar 

  4. Fernandes, F.T., de Oliveira, T.A., Teixeira, C.E., de Moraes Batista, A.F., Dalla Costa, G., Chiavegatto Filho, A.D.P.: A multipurpose machine learning approach to predict covid-19 negative prognosis in São Paulo, Brazil. Sci. Rep. 11(1), 1–7 (2021)

    Article  Google Scholar 

  5. Wynants, L., et al.: Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 369 (2020). https://doi.org/10.1136/bmj.m1328

  6. Seedat, N., Crabbe J., van der Schaar, M.: Data-SUITE: data-centric identification of in-distribution incongruous examples. arXiv preprint arXiv:2202.08836 (2022)

  7. Seedat, N., Crabbe J., Bica, I., van der Schaar, M.: Data-IQ: characterizing subgroups with heterogeneous outcomes in tabular data. arXiv preprint arXiv:2210.13043 (2022)

  8. Paiva, P.Y.A., Moreno, C.C., Smith-Miles, K., Valeriano, M.G., Lorena, A.C.: Relating instance hardness to classification performance in a dataset: a visual approach. Mach. Learn., 1–39 (2022)

    Google Scholar 

  9. Smith, M.R., Martinez, T., Giraud-Carrier, C.: An instance level analysis of data complexity. Mach. Learn. 95(2), 225–256 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  10. Arruda, J.L.M., Prudêncio, R.B.C., Lorena, A.C.: Measuring instance hardness using data complexity measures. In: Cerri, R., Prati, R.C. (eds.) BRACIS 2020. LNCS (LNAI), vol. 12320, pp. 483–497. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61380-8_33

    Chapter  Google Scholar 

  11. Paiva, P.Y.A., Smith-Miles, K., Valeriano, M.G., Lorena, A.C.: PyHard: a novel tool for generating hardness embeddings to support data-centric analysis. arXiv preprint arXiv:2109.14430 (2021)

  12. Valeriano, M.G., et al.: Let the data speak: analysing data from multiple health centers of the São Paulo metropolitan area for covid-19 clinical deterioration prediction. In: 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pp. 948–951. IEEE (2022)

    Google Scholar 

  13. Zheng, K., Chen, G., Herschel, M., Ngiam, K.Y., Ooi, B.C., Gao, J.: PACE: learning effective task decomposition for human-in-the-loop healthcare delivery. In: Proceedings of the 2021 International Conference on Management of Data, pp. 2156–2168 (2021)

    Google Scholar 

  14. Houston, A., Cosma, G., Turner, P., Bennett, A.: Predicting surgical outcomes for chronic exertional compartment syndrome using a machine learning framework with embedded trust by interrogation strategies. Sci. Rep. 11(1), 1–15 (2021)

    Article  Google Scholar 

  15. Prudêncio, R.B., Silva Filho, T.M.: Explaining learning performance with local performance regions and maximally relevant meta-rules. In: Xavier-Junior, J.C., Rios, R.A. (eds.) Brazilian Conference on Intelligent Systems, pp. 550–564. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-21686-2_38

  16. Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., Yang, G.-Z.: XAI-explainable artificial intelligence. Sci. Rob. 4(37), eaay7120 (2019)

    Google Scholar 

  17. Ojala, M., Garriga, G.C.: Permutation tests for studying classifier performance. J. Mach. Learn. Res. 11(6) (2010)

    Google Scholar 

  18. Ghorbani, A., Zou, J.: Data Shapley: equitable valuation of data for machine learning. In: International Conference on Machine Learning, pp. 2242–2251. PMLR (2019)

    Google Scholar 

  19. Lorena, A.C., Garcia, L.P., Lehmann, J., Souto, M.C., Ho, T.K.: How complex is your classification problem? A survey on measuring classification complexity. ACM Comput. Surv. 52(5), 1–34 (2019)

    Article  Google Scholar 

  20. Jafarzadeh, A., Jafarzadeh, S., Nozari, P., Mokhtari, P., Nemati, M.: Lymphopenia an important immunological abnormality in patients with covid-19: possible mechanisms. Scand. J. Immunol. 93(2), e12967 (2021)

    Article  Google Scholar 

  21. Amankwaa-Kyeremeh, B., Greet, C., Zanin, M., Skinner, W., Asamoah, R.K.: Selecting key predictor parameters for regression analysis using modified Neighbourhood Component Analysis (NCA) algorithm. In: Proceedings of 6th UMaT Biennial International Mining and Mineral Conference, pp. 320–325 (2020)

    Google Scholar 

  22. Smith-Miles, K., Tan, T.T.: Measuring algorithm footprints in instance space. In: 2012 IEEE Congress on Evolutionary Computation, pp. 1–8. IEEE (2012)

    Google Scholar 

  23. Muñoz, M.A., Villanova, L., Baatar, D., Smith-Miles, K.: Instance spaces for machine learning classification. Mach. Learn. 107(1), 109–147 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  24. Khan, K., Rehman, S.U., Aziz, K., Fong, S., Sarasvady, S.: DBSCAN: past, present and future. In: The Fifth International Conference on the Applications of Digital Information and Web Technologies, pp. 232–238. IEEE (2014)

    Google Scholar 

  25. Edelsbrunner, H.: Alpha shapes-a survey. Tessellations Sci. 27, 1–25 (2010)

    Google Scholar 

  26. Najafabadi, M.M., Villanustre, F., Khoshgoftaar, T.M., Seliya, N., Wald, R., Muharemagic, E.: Deep learning applications and challenges in big data analytics. J. Big Data 2(1), 1–21 (2015)

    Article  Google Scholar 

Download references

Acknowledgements

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001. The authors also thank the financial support of FAPESP (grant 2021/06870-3) and CNPq.

Author information

Authors and Affiliations

  1. Instituto Tecnológico de Aeronáutica, Praça Marechal Eduardo Gomes, São José dos Campos, Brazil

    Maria Gabriela Valeriano, Pedro Yuri Arbs Paiva & Ana Carolina Lorena

  2. Universidade Federal de São Paulo, Rua Botucatu, São Paulo, Brazil

    Maria Gabriela Valeriano & Carlos Roberto Veiga Kiffer

Authors
  1. Maria Gabriela Valeriano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pedro Yuri Arbs Paiva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carlos Roberto Veiga Kiffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ana Carolina Lorena
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Gabriela Valeriano .

Editor information

Editors and Affiliations

  1. Federal University of São Carlos, São Carlos, Brazil

    Murilo C. Naldi

  2. Centro Universitario da FEI, São Bernardo do Campo, Brazil

    Reinaldo A. C. Bianchi

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Valeriano, M.G., Paiva, P.Y.A., Kiffer, C.R.V., Lorena, A.C. (2023). A Framework for Characterizing What Makes an Instance Hard to Classify. In: Naldi, M.C., Bianchi, R.A.C. (eds) Intelligent Systems. BRACIS 2023. Lecture Notes in Computer Science(), vol 14196. Springer, Cham. https://doi.org/10.1007/978-3-031-45389-2_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-45389-2_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-45388-5

  • Online ISBN: 978-3-031-45389-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation