Design and Implementation of a DataOps-Based Asynchronous Pipeline for Large-Scale Data Ingestion - Amorim, Leonardo Afonso; Aguiar, Vinicius Alboneti; Oliveira, Sávio Salvarino Teles de; Galvão Filho, Arlindo Rodrigues; Costa, Ricardo; Prado, Marcos (Anais da Escola Regional de Informática de Goiás (ERI-GO); 2025: Anais da XIII Escola Regional de Informática de GoiásDO - 10.5753/erigo.2025.17069, 2025)
This paper presents the design and validation of a data operations (DataOps)-oriented asynchronous pipeline designed to optimize large-scale data ingestion into cloud data warehouses. The proposed solution addresses ingestion bottlenecks caused by large file transfers through a combination of strategic file fragmentation and parallel processing using serverless services, including Google Cloud Functions and Cloud Run. Rather than introducing a new architectural model, the contribution lies in demonstrating the practical effectiveness of integrating these cloud-native techniques under real-world conditions. Experimental results show a 52.8x speed-up in ingestion time, reducing the processing of a 2TB dataset from 11 days to just 5 hours. The pipeline also incorporates Infrastructure as Code (IaC), Continuous Integration/Continuous Deployment (CI/CD), and automated monitoring, delivering a scalable, reproducible, and cost-efficient ingestion strategy well-suited for real-world enterprise data workflows.