Preventing Out-Of-Memory Errors in Dask through Automated Memory-Aware Chunking - Fonseca, Daniel De Lucca; Trujillo, Carlos Alberto Astudillo; Borin, Edson (Anais do Simpósio em Sistemas Computacionais de Alto Desempenho (SSCAD), 2025)
Data-parallel frameworks like Dask partition datasets into chunks for concurrent execution, but choosing suitable chunk dimensions remains challenging: oversized chunks cause Out-Of-Memory (OOM) failures while undersized chunks reduce performance. This paper introduces memory-aware chunking that predicts peak memory from input shapes using linear regression and automatically derives optimal chunk sizes adapted to each operator’s memory requirements. Evaluation on seismic imaging operators across 768 trials shows complete elimination of OOM failures (versus 31.6% failure rate for Dask’s default chunking) and 52% peak memory reduction, enabling reliable distributed processing in memory-constrained environments.