Horizontal vs. Vertical Executor Scaling for TPC-DS Analytical Workloads on Geo-Distributed Kubernetes - Guimaraes, Italo V. P.; Araujo, Aleteia (Anais do Seminário Integrado de Software e Hardware (SEMISH), 2026)
When deploying Apache Spark on Kubernetes, practitioners must decide whether to allocate resources as many small executors or as few large ones. The conventional argument for large executors that they reduce shuffle redistribution rests on an assumption that Adaptive Query Execution (AQE) in Spark 3.5 renders false: shuffle volume is virtually identical across all executor sizes, because AQE adapts partition plans to the data at runtime, not to executor configuration. Without a shuffle advantage, vertical scaling only adds JVM garbage-collection overhead: large executors accumulate up to 84% more GC time, and horizontal scaling delivers 22% shorter execution at the same resource budget. Practitioners should prefer small executors (4 cores, 14 GB) and scale out.