Large-Scale Health Data Pipelines with a Lakehouse Architecture: A SISVAN Case Study

• Italo V. P. Guimaraes UnB https://orcid.org/0009-0000-2291-1807
• Marcelo Ladeira UnB https://orcid.org/0000-0003-1542-6293
• Aleteia Araujo UnB https://orcid.org/0000-0003-4645-6700

DOI: https://doi.org/10.5753/semish.2026.23844

Resumo

National health surveillance systems such as Brazil’s SISVAN accumulate tens of millions of records yet still rely on rigid data-warehouse pipelines ill-suited to heterogeneous, evolving data. This paper shows that a Lakehouse architecture — Delta Lake over MinIO, orchestrated by Apache Airflow and queried via Trino — processes 82 GB of SISVAN microdata through four medallion layers in under 100 minutes and returns Gold-layer analytical queries in under 2 seconds. A Superset dashboard co-designed with Ministry of Health analysts translates the curated data into actionable public-health indicators. The results validate that modern Lakehouse patterns deliver ACID guarantees, full data lineage, and sub-second BI latency on commodity hardware for national-scale health data governance.

Referências

Armbrust, M., Das, T., Sun, L., Yavuz, B., Zhu, S., Murthy, M., Torres, J., van Hovell, H., Ionescu, A., et al. (2020). Delta lake: high-performance acid table storage over cloud object stores. Proc. VLDB Endow., 13(12):3411–3424.

Begoli, E., Goethert, I., and Knight, K. (2021). A lakehouse architecture for the management and analysis of heterogeneous data for biomedical research and mega-biobanks. In 2021 IEEE International Conference on Big Data (Big Data), pages 4643–4651.

Giebler, C., Gröger, C., Hoos, E., Schwarz, H., and Mitschang, B. (2020). A zone reference model for enterprise-grade data lake management. In 2020 IEEE 24th International Enterprise Distributed Object Computing Conference (EDOC), pages 57–66.

Hai, R., Koutras, C., Quix, C., and Jarke, M. (2023). Data lakes: A survey of functions and systems. IEEE Transactions on Knowledge and Data Engineering, 35(12):12571–12590.

Harby, A. A. and Zulkernine, F. (2022). From data warehouse to lakehouse: A comparative review. In 2022 IEEE International Conference on Big Data (Big Data), pages 389–395.

Janssen, N., Ilayperuma, T., Jayasinghe, J., Bukhsh, F., and Daneva, M. (2024). The evolution of data storage architectures: examining the secure value of the data lakehouse. Journal of Data, Information and Management, 6(4):309–334.

Li, G., Hu, W., and You, T. (2023). Data lake development status and outlook. In Third International Conference on Green Communication, Network, and Internet of Things (CNIoT 2023), volume 12814, page 128142G. SPIE.

Manco, C., Dolci, T., Azzalini, F., Barbierato, E., Gribaudo, M., and Tanca, L. (2023). HEALER: A Data Lake Architecture for Healthcare. DEU.

Patil, S. and Belloum, A. S. Z. A. (2024). Processing fhir in modern data lakehouse. In 2024 IEEE 20th International Conference on e-Science (e-Science), pages 1–7.

Zhao, T., Hai, N., Li, W., Zheng, W., Zhang, Y., Li, X., and Fei, G. (2023). Multi-modal medical data exploration based on data lake. In Health Information Science, pages 213–222. Springer Nature.