Large-Scale Health Data Pipelines with a Lakehouse Architecture: A SISVAN Case Study - Guimaraes, Italo V. P.; Ladeira, Marcelo; Araujo, Aleteia (Anais do Seminário Integrado de Software e Hardware (SEMISH), 2026)
National health surveillance systems such as Brazil’s SISVAN accumulate tens of millions of records yet still rely on rigid data-warehouse pipelines ill-suited to heterogeneous, evolving data. This paper shows that a Lakehouse architecture — Delta Lake over MinIO, orchestrated by Apache Airflow and queried via Trino — processes 82 GB of SISVAN microdata through four medallion layers in under 100 minutes and returns Gold-layer analytical queries in under 2 seconds. A Superset dashboard co-designed with Ministry of Health analysts translates the curated data into actionable public-health indicators. The results validate that modern Lakehouse patterns deliver ACID guarantees, full data lineage, and sub-second BI latency on commodity hardware for national-scale health data governance.