5LTEP-L4: An Open-Source CKAN Toolkit for Provenance-Enabled Observability of Open Government Data
Resumo
We present 5LTEP-L4, an open-source Python toolkit that turns any CKAN-based Open Government Data (OGD) portal into a source of W3C PROV-DM provenance records. Through the CKAN Web API, it fingerprints dataset metadata with SHA-256, classifies each change as a content modification, schema drift, retroactive alteration, or clean update, and publishes PROV-O JSON-LD records in an append-only Git audit trail, orchestrated by GitHub Actions with no dedicated hosting. A 109-day deployment on the IBAMA portal (430 scheduled cycles, 97.2% successful, 79 datasets) detected 31 changes, all confirmed against an independent snapshot diff; controlled fault injection reached 632/632 correct classifications. The toolkit ran unmodified on two further CKAN portals (Recife, 223 datasets; ANEEL, 72), and its records load in the prov reference library. Code is available under the MIT license at https://github.com/lsp3cesarschool/5ltep-layer4.
Palavras-chave:
data provenance, open government data, W3C PROV, linked data, CKAN, observability, GitHub Actions
Referências
Riccardo Albertoni and Antoine Isaac. 2016. Data on the Web Best Practices: Data Quality Vocabulary. W3C Working Group Note. [link]
Judie Attard, Fabrizio Orlandi, Simon Scerri, and Sören Auer. 2015. A Systematic Review of Open Government Data Initiatives. Government Information Quarterly 32, 4 (2015), 399–418. DOI: 10.1016/j.giq.2015.07.006
James Cheney, Laura Chiticariu, and Wang-Chiew Tan. 2009. Provenance in Databases: Why, How, and Where. Foundations and Trends in Databases 1, 4 (2009), 379–474. DOI: 10.1561/1900000006
Juliana Freire, David Koop, Emanuele Santos, and Cláudio T. Silva. 2008. Provenance for Computational Tasks: A Survey. Computing in Science & Engineering 10, 3 (2008), 11–21. DOI: 10.1109/MCSE.2008.79
Bernardo Gallo, Lúcia Drummond, and José Viterbo. 2025. Mitigating Concept Drift in Job Execution Time Prediction Models. In Proceedings of the 13th Symposium on Knowledge Discovery, Mining and Learning (KDMiLe). SBC, Fortaleza, CE, Brazil, 81–88. DOI: 10.5753/kdmile.2025.247724
Trung Dong Huynh. 2026. prov: A Python Library for the W3C Provenance Data Model (version 3.2.2). Software, MIT License. [link]
Timothy Lebo, Satya Sahoo, and Deborah McGuinness. 2013. PROV-O: The PROV Ontology. W3C Recommendation. [link]
Charity Majors, Liz Fong-Jones, and George Miranda. 2022. Observability Engineering: Achieving Production Excellence. O’Reilly Media, Sebastopol, CA.
Luc Moreau and Paolo Missier. 2013. PROV-DM: The PROV Data Model. W3C Recommendation. [link]
Sebastian Neumaier, Jürgen Umbrich, and Axel Polleres. 2016. Automated Quality Assessment of Metadata across Open Data Portals. Journal of Data and Information Quality 8, 1, Article 2 (2016), 29 pages. DOI: 10.1145/2964909
Luciano Santos Pinheiro, Cristiane de Holanda de Barros e Silva, Vitor Barros Aquino, Thays Maria da Conceição Silva Carvalho, Cristiano Vale do Rego Barros Filho, and Washington Henrique Carvalho Almeida. 2026. Towards Trust Engineering in Open Data Systems: A Layered Conceptual Framework Integrating Quality Assurance and Governance Perspectives. In SOFTENG 2026: The Twelfth International Conference on Advances and Trends in Software Engineering. IARIA, Venice, Italy, 21–28. [link]
Presidência da República. 2011. Lei nº 12.527, de 18 de novembro de 2011: Lei de Acesso à Informação. Diário Oficial da União. [link]
Presidência da República. 2016. Decreto nº 8.777, de 11 de maio de 2016: Institui a Política de Dados Abertos do Poder Executivo Federal. Diário Oficial da União. [link]
Fabiana Santos, Lucas Giusti, Diego Carvalho, Eduardo Ogasawara, and Jorge Soares. 2024. Avaliação de Desvios de Conceitos Reais e Virtuais nos Atrasos de Voos em São Paulo nos Períodos Pré, Intra e Pós-Pandemia. In Proceedings of the 39th Brazilian Symposium on Databases (SBBD). SBC, Florianópolis, SC, Brazil, 827–833. DOI: 10.5753/sbbd.2024.243111
Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Biessmann, and Andreas Grafberger. 2018. Automating Large-Scale Data Quality Verification. Proceedings of the VLDB Endowment 11, 12 (2018), 1781–1794. DOI: 10.14778/3229863.3229867
D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems 28 (NeurIPS). Curran Associates, Montréal, Canada, 2503–2511.
Yogesh L. Simmhan, Beth Plale, and Dennis Gannon. 2005. A Survey of Data Provenance in e-Science. ACM SIGMOD Record 34, 3 (2005), 31–36. DOI: 10.1145/1084805.1084812
Jürgen Umbrich, Sebastian Neumaier, and Axel Polleres. 2015. Quality Assessment and Evolution of Open Data Portals. In Proceedings of the 3rd International Conference on Future Internet of Things and Cloud (FiCloud). IEEE, Rome, Italy, 404–411. DOI: 10.1109/FiCloud.2015.82
Antonio Vetrò, Lorenzo Canova, Marco Torchiano, Camilo Orozco Minotas, Raimondo Iemma, and Federico Morando. 2016. Open Data Quality Measurement Framework: Definition and Application to Open Government Data. Government Information Quarterly 33, 2 (2016), 325–337. DOI: 10.1016/j.giq.2016.02.001
Mark D.Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J. G. Gray, Paul Groth, Carole Goble, Jeffrey S. Grethe, Jaap Heringa, Peter A. C. ’t Hoen, Rob Hooft, Tobias Kuhn, Ruben Kok, Joost Kok, Scott J. Lusher, Maryann E. Martone, Albert Mons, Abel L. Packer, Bengt Persson, Philippe Rocca-Serra, Marco Roos, Rene van Schaik, Susanna-Assunta Sansone, Erik Schultes, Thierry Sengstag, Ted Slater, George Strawn, Morris A. Swertz, Mark Thompson, Johan van der Lei, Erik van Mulligen, Jan Velterop, Andra Waagmeester, Peter Wittenburg, Katherine Wolstencroft, Jun Zhao, and Barend Mons. 2016. The FAIR Guiding Principles for Scientific Data Management and Stewardship. Scientific Data 3 (2016), 160018. DOI: 10.1038/sdata.2016.18
Amrapali Zaveri, Anisa Rula, Andrea Maurino, Ricardo Pietrobon, Jens Lehmann, and Sören Auer. 2016. Quality Assessment for Linked Data: A Survey. Semantic Web 7, 1 (2016), 63–93. DOI: 10.3233/SW-150175
Judie Attard, Fabrizio Orlandi, Simon Scerri, and Sören Auer. 2015. A Systematic Review of Open Government Data Initiatives. Government Information Quarterly 32, 4 (2015), 399–418. DOI: 10.1016/j.giq.2015.07.006
James Cheney, Laura Chiticariu, and Wang-Chiew Tan. 2009. Provenance in Databases: Why, How, and Where. Foundations and Trends in Databases 1, 4 (2009), 379–474. DOI: 10.1561/1900000006
Juliana Freire, David Koop, Emanuele Santos, and Cláudio T. Silva. 2008. Provenance for Computational Tasks: A Survey. Computing in Science & Engineering 10, 3 (2008), 11–21. DOI: 10.1109/MCSE.2008.79
Bernardo Gallo, Lúcia Drummond, and José Viterbo. 2025. Mitigating Concept Drift in Job Execution Time Prediction Models. In Proceedings of the 13th Symposium on Knowledge Discovery, Mining and Learning (KDMiLe). SBC, Fortaleza, CE, Brazil, 81–88. DOI: 10.5753/kdmile.2025.247724
Trung Dong Huynh. 2026. prov: A Python Library for the W3C Provenance Data Model (version 3.2.2). Software, MIT License. [link]
Timothy Lebo, Satya Sahoo, and Deborah McGuinness. 2013. PROV-O: The PROV Ontology. W3C Recommendation. [link]
Charity Majors, Liz Fong-Jones, and George Miranda. 2022. Observability Engineering: Achieving Production Excellence. O’Reilly Media, Sebastopol, CA.
Luc Moreau and Paolo Missier. 2013. PROV-DM: The PROV Data Model. W3C Recommendation. [link]
Sebastian Neumaier, Jürgen Umbrich, and Axel Polleres. 2016. Automated Quality Assessment of Metadata across Open Data Portals. Journal of Data and Information Quality 8, 1, Article 2 (2016), 29 pages. DOI: 10.1145/2964909
Luciano Santos Pinheiro, Cristiane de Holanda de Barros e Silva, Vitor Barros Aquino, Thays Maria da Conceição Silva Carvalho, Cristiano Vale do Rego Barros Filho, and Washington Henrique Carvalho Almeida. 2026. Towards Trust Engineering in Open Data Systems: A Layered Conceptual Framework Integrating Quality Assurance and Governance Perspectives. In SOFTENG 2026: The Twelfth International Conference on Advances and Trends in Software Engineering. IARIA, Venice, Italy, 21–28. [link]
Presidência da República. 2011. Lei nº 12.527, de 18 de novembro de 2011: Lei de Acesso à Informação. Diário Oficial da União. [link]
Presidência da República. 2016. Decreto nº 8.777, de 11 de maio de 2016: Institui a Política de Dados Abertos do Poder Executivo Federal. Diário Oficial da União. [link]
Fabiana Santos, Lucas Giusti, Diego Carvalho, Eduardo Ogasawara, and Jorge Soares. 2024. Avaliação de Desvios de Conceitos Reais e Virtuais nos Atrasos de Voos em São Paulo nos Períodos Pré, Intra e Pós-Pandemia. In Proceedings of the 39th Brazilian Symposium on Databases (SBBD). SBC, Florianópolis, SC, Brazil, 827–833. DOI: 10.5753/sbbd.2024.243111
Sebastian Schelter, Dustin Lange, Philipp Schmidt, Meltem Celikel, Felix Biessmann, and Andreas Grafberger. 2018. Automating Large-Scale Data Quality Verification. Proceedings of the VLDB Endowment 11, 12 (2018), 1781–1794. DOI: 10.14778/3229863.3229867
D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems 28 (NeurIPS). Curran Associates, Montréal, Canada, 2503–2511.
Yogesh L. Simmhan, Beth Plale, and Dennis Gannon. 2005. A Survey of Data Provenance in e-Science. ACM SIGMOD Record 34, 3 (2005), 31–36. DOI: 10.1145/1084805.1084812
Jürgen Umbrich, Sebastian Neumaier, and Axel Polleres. 2015. Quality Assessment and Evolution of Open Data Portals. In Proceedings of the 3rd International Conference on Future Internet of Things and Cloud (FiCloud). IEEE, Rome, Italy, 404–411. DOI: 10.1109/FiCloud.2015.82
Antonio Vetrò, Lorenzo Canova, Marco Torchiano, Camilo Orozco Minotas, Raimondo Iemma, and Federico Morando. 2016. Open Data Quality Measurement Framework: Definition and Application to Open Government Data. Government Information Quarterly 33, 2 (2016), 325–337. DOI: 10.1016/j.giq.2016.02.001
Mark D.Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J. G. Gray, Paul Groth, Carole Goble, Jeffrey S. Grethe, Jaap Heringa, Peter A. C. ’t Hoen, Rob Hooft, Tobias Kuhn, Ruben Kok, Joost Kok, Scott J. Lusher, Maryann E. Martone, Albert Mons, Abel L. Packer, Bengt Persson, Philippe Rocca-Serra, Marco Roos, Rene van Schaik, Susanna-Assunta Sansone, Erik Schultes, Thierry Sengstag, Ted Slater, George Strawn, Morris A. Swertz, Mark Thompson, Johan van der Lei, Erik van Mulligen, Jan Velterop, Andra Waagmeester, Peter Wittenburg, Katherine Wolstencroft, Jun Zhao, and Barend Mons. 2016. The FAIR Guiding Principles for Scientific Data Management and Stewardship. Scientific Data 3 (2016), 160018. DOI: 10.1038/sdata.2016.18
Amrapali Zaveri, Anisa Rula, Andrea Maurino, Ricardo Pietrobon, Jens Lehmann, and Sören Auer. 2016. Quality Assessment for Linked Data: A Survey. Semantic Web 7, 1 (2016), 63–93. DOI: 10.3233/SW-150175
Publicado
27/10/2026
Como Citar
PINHEIRO, Luciano Santos; SÉRGIO, Anderson Tenório.
5LTEP-L4: An Open-Source CKAN Toolkit for Provenance-Enabled Observability of Open Government Data. In: WORKSHOP DE FERRAMENTAS E APLICAÇÕES - SIMPÓSIO BRASILEIRO DE SISTEMAS MULTIMÍDIA E WEB (WEBMEDIA), 32. , 2026, Lavras/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 82-85.
ISSN 2596-1683.
DOI: https://doi.org/10.5753/webmedia_estendido.2026.33713.
