MALiBU: Metaheuristics Approach for Online Load Balancing in MapReduce with Skewed Data Input

• Matheus H. M. Pericini
• Lucas G. M. Leite
• Javam C. Machado

Abstract

MapReduce is a parallel computing model where a large dataset is split into smaller parts and executed on multiple machines. When data are not uniformly distributed, we have the so called partitioning skew, where the allocation of tasks to machines becomes unbalanced, either by the distribution function splitting the dataset unevenly or because a part of the data is more complex and requires greater computing effort. To solve this problem, we propose a function based on Simulated Annealing metaheuristic which finds a partitioning that results in a better load balancing.