Avaliação Comparativa de Técnicas baseadas em Deep Learning para Identificação de Duplicatas
Resumo
Dados de aplicações possuem inevitavelmente inconsistências que podem causar mal funcionamento em operações rotineiras e compromenter resultados analíticos. Um tipo particular de inconsistência é a presença de duplicatas, isto é, múltiplas e não idênticas representações da mesma informação. A identificação de duplicatas é difícil porque elas não são cópias idênticas entre si. Recentemente, DeepMatcher e Ditto, duas soluções baseadas em deep learning, obtiveram resultados do estado da arte em identificação de duplicatas. Entretanto, DeepMatcher e Ditto não consideraram duplicatas com variações textuais em nível de caracteres em seus experimentos; tais variações são pervasivas em bancos de dados do mundo real. Este artigo apresenta uma avaliação comparativa do DeepMatcher e Ditto em dados com padrões textuais que não foram considerados nos experimentos anteriores. Os resultados obtidos demonstraram que as duas soluções experimentaram queda de acurácia, sendo que o Ditto apresentou maior robustez em comparação com DeepMatcher.
Referências
Abedjan, Z., Chu, X., Deng, D., Fernandez, R. C., Ilyas, I. F., Ouzzani, M., Papotti, P., Stonebraker, M., and Tang, N. (2016). Detecting Data Errors: Where Are We and What Needs to Be Done? Proceedings of the VLDB Endowment, 9(12):993–1004.
Ananthakrishna, R., Chaudhuri, S., and Ganti, V. (2002). Eliminating Fuzzy Duplicates in Data Warehouses. In Proceedings of the VLDB Conference, pages 586–597.
Barlaug, N. and Gulla, J. A. (2021). Neural Networks for Entity Matching: A Survey. ACM Transactions on Knowledge Discovery from Data, 15(3):52:1–52:37.
Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135–146.
Brunner, U. and Stockinger, K. (2020). Entity Matching with Transformer Architectures – A Step Forward in Data Integration. In Proceedings of the International Conference on Extending Database Technology, pages 463–473.
Clark, K. and Manning, C. D. (2016). Improving Coreference Resolution by Learning Entity-Level Distributed Representations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 643–653.
de Oliveira, J. D. and Ribeiro, L. A. (2019). Avaliação de Processos ETL para Análise de Dados usando SGBD Orientado a Grafos. In Anais da VII Escola Regional de Informática de Goiás, pages 61–74.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186.
Ebraheem, M., Thirumuruganathan, S., Joty, S. R., Ouzzani, M., and Tang, N. (2018). Distributed Representations of Tuples for Entity Resolution. Proceedings of the VLDB Endowment, 11(11):1454–1467.
Elmagarmid, A. K., Ipeirotis, P. G., and Verykios, V. S. (2007). Duplicate Record Detection: A Survey. IEEE Transactions on Knowledge and Data Engineering, 19(1):1–16.
Hernández, M. A. and Stolfo, S. J. (1998). Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem. Data Mining and Knowledge Discovery, 2(1):9–37.
Konda, P., Das, S., C., P. S. G., Doan, A., Ardalan, A., Ballard, J. R., Li, H., Panahi, F., Zhang, H., Naughton, J. F., Prasad, S., Krishnan, G., Deep, R., and Raghavendra, V. (2016). Magellan: Toward Building Entity Matching Management Systems. Proceedings of the VLDB Endowment, 9(12):1197–1208.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the Conference on Neural Information Processing Systems, pages 1106–1114.
LeCun, Y., Bengio, Y., and Hinton, G. E. (2015). Deep Learning. Nature, 521(7553):436–444.
Leone, M., Huber, S., Arora, A., García-Durán, A., and West, R. (2022). A Critical Re-evaluation of Neural Methods for Entity Alignment. Proceedings of the VLDB Endowment, 15(8):1712–1725.
Li, Y., Li, J., Suhara, Y., Doan, A., and Tan, W. (2020). Deep Entity Matching with Pre-Trained Language Models. Proceedings of the VLDB Endowment, 14(1):50–60.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692.
Mudgal, S., Li, H., Rekatsinas, T., Doan, A., Park, Y., Krishnan, G., Deep, R., Arcaute, E., and Raghavendra, V. (2018). Deep Learning for Entity Matching: A Design Space Exploration. In Proceedings of the SIGMOD Conference, pages 19–34. ACM.
Newcombe, H., Kennedy, J., Axford, S., and James, A. (1959). Automatic Linkage of Vital Records. Science, 130(3381):954–959.
Shen, W., Wang, J., and Han, J. (2015). Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions. IEEE Transactions on Knowledge and Data Engineering, 27(2):443–460.
Stonebraker, M. and Ilyas, I. F. (2018). Data Integration: The Current Status and the Way Forward. IEEE Data Engineering Bulletin, 41(2):3–9.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is All you Need. In Proceedings of the Conference on Neural Information Processing Systems, pages 5998–6008.
Ananthakrishna, R., Chaudhuri, S., and Ganti, V. (2002). Eliminating Fuzzy Duplicates in Data Warehouses. In Proceedings of the VLDB Conference, pages 586–597.
Barlaug, N. and Gulla, J. A. (2021). Neural Networks for Entity Matching: A Survey. ACM Transactions on Knowledge Discovery from Data, 15(3):52:1–52:37.
Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135–146.
Brunner, U. and Stockinger, K. (2020). Entity Matching with Transformer Architectures – A Step Forward in Data Integration. In Proceedings of the International Conference on Extending Database Technology, pages 463–473.
Clark, K. and Manning, C. D. (2016). Improving Coreference Resolution by Learning Entity-Level Distributed Representations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 643–653.
de Oliveira, J. D. and Ribeiro, L. A. (2019). Avaliação de Processos ETL para Análise de Dados usando SGBD Orientado a Grafos. In Anais da VII Escola Regional de Informática de Goiás, pages 61–74.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186.
Ebraheem, M., Thirumuruganathan, S., Joty, S. R., Ouzzani, M., and Tang, N. (2018). Distributed Representations of Tuples for Entity Resolution. Proceedings of the VLDB Endowment, 11(11):1454–1467.
Elmagarmid, A. K., Ipeirotis, P. G., and Verykios, V. S. (2007). Duplicate Record Detection: A Survey. IEEE Transactions on Knowledge and Data Engineering, 19(1):1–16.
Hernández, M. A. and Stolfo, S. J. (1998). Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem. Data Mining and Knowledge Discovery, 2(1):9–37.
Konda, P., Das, S., C., P. S. G., Doan, A., Ardalan, A., Ballard, J. R., Li, H., Panahi, F., Zhang, H., Naughton, J. F., Prasad, S., Krishnan, G., Deep, R., and Raghavendra, V. (2016). Magellan: Toward Building Entity Matching Management Systems. Proceedings of the VLDB Endowment, 9(12):1197–1208.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the Conference on Neural Information Processing Systems, pages 1106–1114.
LeCun, Y., Bengio, Y., and Hinton, G. E. (2015). Deep Learning. Nature, 521(7553):436–444.
Leone, M., Huber, S., Arora, A., García-Durán, A., and West, R. (2022). A Critical Re-evaluation of Neural Methods for Entity Alignment. Proceedings of the VLDB Endowment, 15(8):1712–1725.
Li, Y., Li, J., Suhara, Y., Doan, A., and Tan, W. (2020). Deep Entity Matching with Pre-Trained Language Models. Proceedings of the VLDB Endowment, 14(1):50–60.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.11692.
Mudgal, S., Li, H., Rekatsinas, T., Doan, A., Park, Y., Krishnan, G., Deep, R., Arcaute, E., and Raghavendra, V. (2018). Deep Learning for Entity Matching: A Design Space Exploration. In Proceedings of the SIGMOD Conference, pages 19–34. ACM.
Newcombe, H., Kennedy, J., Axford, S., and James, A. (1959). Automatic Linkage of Vital Records. Science, 130(3381):954–959.
Shen, W., Wang, J., and Han, J. (2015). Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions. IEEE Transactions on Knowledge and Data Engineering, 27(2):443–460.
Stonebraker, M. and Ilyas, I. F. (2018). Data Integration: The Current Status and the Way Forward. IEEE Data Engineering Bulletin, 41(2):3–9.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is All you Need. In Proceedings of the Conference on Neural Information Processing Systems, pages 5998–6008.
Publicado
25/10/2022
Como Citar
LIMA, Paulo Henrique Santos; MARTINS, Wellington Santos; RIBEIRO, Leonardo Andrade.
Avaliação Comparativa de Técnicas baseadas em Deep Learning para Identificação de Duplicatas. In: ESCOLA REGIONAL DE INFORMÁTICA DE GOIÁS (ERI-GO), 10. , 2022, Goiás.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 153-164.
DOI: https://doi.org/10.5753/erigo.2022.227704.
