Reconciliação de dados usando MapReduce
Resumo
Um dos principais desafios na área de integração de dados é o grande volume de dados gerado por aplicações atuais. Neste artigo são propostas funções de Map e Reduce para a integração de dados utilizando de dados de procedência. Essas funções são baseadas em políticas encontradas na literatura e, aplicadas no contexto distribuído. Como resultado, obteve-se um melhor tempo de execução com o Hadoop MapReduce, em comparação com a execução centralizada das funções, mesmo considerando um volume pequeno de dados nos experimentos. Além disso, as funções se mostraram eficazes de acordo com a política considerada, em ambos os ambientes.
Palavras-chave:
MapReduce, reconciliação de dados, integração de dados
Referências
Almeida, D. S.; Hara, C. S., Ciferri, R. R., and Ciferri, C. D. A. (2018). An asynchronous collaborative reconciliation model based on data provenance. Software: Pratice and- Experience, 48(1):197–232.
Apache Software Foundation (2025). Apache hadoop. [link]. Acessado em: 13 jun. 2025.
Azuan, N. A. A. (2021). Exploring Manual Correction as a Source of User Feedback in Pay-As-You-Go Integration. PhD thesis, The University of Manchester.
Bhandarkar, M. (2010). Mapreduce programming with apache hadoop. In 2010 IEEE International Symposium on Parallel Distributed Processing (IPDPS), pages 1–1.
Cheney, J., Chiticariu, L., Tan, W.-C., et al. (2009). Provenance in databases: Why, how, and where. Foundations and Trends® in Databases, 1(4):379–474.
Dean, J.; Ghemawat, S. (2008). Mapreduce: simplified data processing on large clusters. Communications of the ACM - 50th anniversary issue: 1958 - 2008, 51(1):107–113.
Khan, W., Kumar, T., Zhang, C., Raj, K., Roy, A. M., and Luo, B. (2023). Sql and nosql database software architecture performance analysis and assessments—a systematic literature review. Big Data and Cognitive Computing, 7(2).
Kruse, F., Hassan, A. P., Awick, J.-P., Gómez, J. M., and Bui, T. (2020). A qualitative literature review on linkage techniques for data integration. In HICSS, pages 1–11.
Mahmood, T., Jami, S. I., Shaikh, Z. A., and Mughal, M. H. (2013). Toward the modeling of data provenance in scientific publications. Computer Standards & Interfaces, 35(1):6–29.
Shrestha, S. and Bhalla, S. (2020). Survey on the evolution of models of data integration. Int. J. Knowl. Based Comput. Syst, 8:11–16.
Stojanović, A., Horvat, M., and Kovačević, Z. (2022). An overview of data integration principles for heterogeneous databases. In 2022 45th Jubilee International Convention on Information, Communication and Electronic Technology (MIPRO), pages 1111–1116.
Yousif, O., Zakaria, R., Aminudin, E., Yahya, K., Sam, A., Singaram, L., Munikanan, V., Yahya, M., Wahi, N., and Shamsuddin, S. (2021). Review of big data integration in construction industry digitalization. Frontiers in Built Environment, 7.
Zheng, L., Pan, J., and Zhang, K. (2022). Power data integration method based on database-table metadata semantic. Journal of Physics: Conference Series, 2179(1):012028.
Apache Software Foundation (2025). Apache hadoop. [link]. Acessado em: 13 jun. 2025.
Azuan, N. A. A. (2021). Exploring Manual Correction as a Source of User Feedback in Pay-As-You-Go Integration. PhD thesis, The University of Manchester.
Bhandarkar, M. (2010). Mapreduce programming with apache hadoop. In 2010 IEEE International Symposium on Parallel Distributed Processing (IPDPS), pages 1–1.
Cheney, J., Chiticariu, L., Tan, W.-C., et al. (2009). Provenance in databases: Why, how, and where. Foundations and Trends® in Databases, 1(4):379–474.
Dean, J.; Ghemawat, S. (2008). Mapreduce: simplified data processing on large clusters. Communications of the ACM - 50th anniversary issue: 1958 - 2008, 51(1):107–113.
Khan, W., Kumar, T., Zhang, C., Raj, K., Roy, A. M., and Luo, B. (2023). Sql and nosql database software architecture performance analysis and assessments—a systematic literature review. Big Data and Cognitive Computing, 7(2).
Kruse, F., Hassan, A. P., Awick, J.-P., Gómez, J. M., and Bui, T. (2020). A qualitative literature review on linkage techniques for data integration. In HICSS, pages 1–11.
Mahmood, T., Jami, S. I., Shaikh, Z. A., and Mughal, M. H. (2013). Toward the modeling of data provenance in scientific publications. Computer Standards & Interfaces, 35(1):6–29.
Shrestha, S. and Bhalla, S. (2020). Survey on the evolution of models of data integration. Int. J. Knowl. Based Comput. Syst, 8:11–16.
Stojanović, A., Horvat, M., and Kovačević, Z. (2022). An overview of data integration principles for heterogeneous databases. In 2022 45th Jubilee International Convention on Information, Communication and Electronic Technology (MIPRO), pages 1111–1116.
Yousif, O., Zakaria, R., Aminudin, E., Yahya, K., Sam, A., Singaram, L., Munikanan, V., Yahya, M., Wahi, N., and Shamsuddin, S. (2021). Review of big data integration in construction industry digitalization. Frontiers in Built Environment, 7.
Zheng, L., Pan, J., and Zhang, K. (2022). Power data integration method based on database-table metadata semantic. Journal of Physics: Conference Series, 2179(1):012028.
Publicado
29/09/2025
Como Citar
SILVA, Adriano L. Leão da; ALMEIDA, Dayse S. de.
Reconciliação de dados usando MapReduce. In: WORKSHOP DE TRABALHOS DE ALUNOS DA GRADUAÇÃO (WTAG) - SIMPÓSIO BRASILEIRO DE BANCO DE DADOS (SBBD), 40. , 2025, Fortaleza/CE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 36-42.
DOI: https://doi.org/10.5753/sbbd_estendido.2025.247720.
