A Preemptive Fair Scheduler Policy for Disco MapReduce Framework
Resumo
Disco is an open source MapReduce framework and an alternative to Hadoop. Preemption of tasks is an important feature which helps organizations relying on the MapReduce paradigm to handle their heterogeneous workload usually constituted of research (long duration and with low priority) and production (short duration and with high priority) applications. The missing preemption in Disco affects the production jobs when these two kinds of jobs need to be executed in parallel: the high priority response is delayed because there aren’t resources to compute it. In this paper we describe the implementation of the Preemptive Fair Scheduler Policy which improved largely our experimental production job execution time with a small impact on the research job.
Referências
Apache Hadoop Website. http://hadoop.apache.org/. Accessed: 2016-04-07.
Armstrong, J. (2003). Making reliable distributed systems in the presence of software errors.
Cheng, L., Zhang, Q., and Boutaba, R. (2011). Mitigating the negative impact of preemption on heterogeneous mapreduce workloads. In Proceedings of the 7th International Conference on Network and Services Management, CNSM ’11, pages 189–197, Laxenburg, Austria, Austria. International Federation for Information Processing.
Cho, B., Rahman, M., Chajed, T., Gupta, I., Abad, C., Roberts, N., and Lin, P. (2013). Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. In Proceedings of the 4th Annual Symposium on Cloud Computing, SOCC ’13, pages 6:1–6:17, New York, NY, USA. ACM.
Cloc Project at Github.com. https://github.com/AlDanial/cloc. Accessed: 2016-04-07.
Dean, J. and Ghemawat, S. (2004). MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of the 6th Conference on Symposium on Operating Systems Design & Implementation - Volume 6, OSDI’04, pages 10–10, Berkeley, CA, USA. USENIX Association.
Digital Ocean Website. https://www.digitalocean.com. Accessed: m2016-04-07.
Disco at Github.com. https://github.com/discoproject/disco. Accessed: 2016-04-07.
Disco Distributed Filesystem Website. http://disco.readthedocs.org/en/latest/howto/ddfs.html. Accessed: 2016-04-07.
Disco Fork at Github.com. https://github.com/discoproject/disco. nAccessed: 2016-04-07.
Ghemawat, S., Gobioff, H., and Leung, S.-T. (2003). The Google File System. In Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP mn’03, pages 29–43, New York, NY, USA. ACM.
Go Worker at Github.com. https://github.com/discoproject/goworker. nAccessed: 2016-04-07.
Haskell Worker at Github.com. https://github.com/zuzia/haskell_worker. Accessed: 2016-04-07.
LFE Worker at Github.com. https://github.com/oubiwann/lfe-disco. Accessed: 2016-04-07.
MAPREDUCE-4584. Umbrella: Preemption and restart of MapReduce tasks - JIRA. https://issues.apache.org/jira/browse/MAPREDUCE-4584. Accessed: 2016-04-07.
MAPREDUCE-5269. Preemption of Reducer (and Shuffle) via checkpointing - JIRA. https://issues.apache.org/jira/browse/MAPREDUCE-5269. Accessed: 2016-04-07.
Mundkur, P., Tuulos, V., and Flatow, J. (2011). Disco: A computing platform for large-scale data analytics. In Proceedings of the 10th ACM SIGPLAN Workshop on Erlang, Erlang ’11, pages 84–89, New York, NY, USA. ACM.
OCamlWorker at Github.com. https://github.com/discoproject/odisco. Accessed: 2016-04-07.
Shvachko, K., Kuang, H., Radia, S., and Chansler, R. (2010). The Hadoop Distributed File System. In Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium on, pages 1–10.
Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., Saha, B., Curino, C., O’Malley, O., Radia, S., Reed, B., and Baldeschwieler, E. (2013). Apache Hadoop YARN: Yet Another Resource Negotiator. In Proceedings of the 4th Annual Symposium on Cloud Computing, SOCC ’13, pages 5:1–5:16, New York, NY, USA. ACM.
Wang, Y., Tan, J., Yu,W., Zhang, L., Meng, X., and Li, X. (2013). Preemptive reducetask scheduling for fair and fast job completion. In Proceedings of the 10th International Conference on Autonomic Computing (ICAC 13), pages 279–289, San Jose, CA. USENIX.
Zaharia, M., Borthakur, D., Sen Sarma, J., Elmeleegy, K., Shenker, S., and Stoica, I. (2010). Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of the 5th European Conference on Computer Systems, EuroSys ’10, pages 265–278, New York, NY, USA. ACM.
Zaharia, M., Konwinski, A., Joseph, A. D., Katz, R., and Stoica, I. (2008). Improving mapreduce performance in heterogeneous environments. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI’08,n pages 29–42, Berkeley, CA, USA. USENIX Association.
Armstrong, J. (2003). Making reliable distributed systems in the presence of software errors.
Cheng, L., Zhang, Q., and Boutaba, R. (2011). Mitigating the negative impact of preemption on heterogeneous mapreduce workloads. In Proceedings of the 7th International Conference on Network and Services Management, CNSM ’11, pages 189–197, Laxenburg, Austria, Austria. International Federation for Information Processing.
Cho, B., Rahman, M., Chajed, T., Gupta, I., Abad, C., Roberts, N., and Lin, P. (2013). Natjam: Design and evaluation of eviction policies for supporting priorities and deadlines in mapreduce clusters. In Proceedings of the 4th Annual Symposium on Cloud Computing, SOCC ’13, pages 6:1–6:17, New York, NY, USA. ACM.
Cloc Project at Github.com. https://github.com/AlDanial/cloc. Accessed: 2016-04-07.
Dean, J. and Ghemawat, S. (2004). MapReduce: Simplified Data Processing on Large Clusters. In Proceedings of the 6th Conference on Symposium on Operating Systems Design & Implementation - Volume 6, OSDI’04, pages 10–10, Berkeley, CA, USA. USENIX Association.
Digital Ocean Website. https://www.digitalocean.com. Accessed: m2016-04-07.
Disco at Github.com. https://github.com/discoproject/disco. Accessed: 2016-04-07.
Disco Distributed Filesystem Website. http://disco.readthedocs.org/en/latest/howto/ddfs.html. Accessed: 2016-04-07.
Disco Fork at Github.com. https://github.com/discoproject/disco. nAccessed: 2016-04-07.
Ghemawat, S., Gobioff, H., and Leung, S.-T. (2003). The Google File System. In Proceedings of the nineteenth ACM symposium on Operating systems principles, SOSP mn’03, pages 29–43, New York, NY, USA. ACM.
Go Worker at Github.com. https://github.com/discoproject/goworker. nAccessed: 2016-04-07.
Haskell Worker at Github.com. https://github.com/zuzia/haskell_worker. Accessed: 2016-04-07.
LFE Worker at Github.com. https://github.com/oubiwann/lfe-disco. Accessed: 2016-04-07.
MAPREDUCE-4584. Umbrella: Preemption and restart of MapReduce tasks - JIRA. https://issues.apache.org/jira/browse/MAPREDUCE-4584. Accessed: 2016-04-07.
MAPREDUCE-5269. Preemption of Reducer (and Shuffle) via checkpointing - JIRA. https://issues.apache.org/jira/browse/MAPREDUCE-5269. Accessed: 2016-04-07.
Mundkur, P., Tuulos, V., and Flatow, J. (2011). Disco: A computing platform for large-scale data analytics. In Proceedings of the 10th ACM SIGPLAN Workshop on Erlang, Erlang ’11, pages 84–89, New York, NY, USA. ACM.
OCamlWorker at Github.com. https://github.com/discoproject/odisco. Accessed: 2016-04-07.
Shvachko, K., Kuang, H., Radia, S., and Chansler, R. (2010). The Hadoop Distributed File System. In Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium on, pages 1–10.
Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., Saha, B., Curino, C., O’Malley, O., Radia, S., Reed, B., and Baldeschwieler, E. (2013). Apache Hadoop YARN: Yet Another Resource Negotiator. In Proceedings of the 4th Annual Symposium on Cloud Computing, SOCC ’13, pages 5:1–5:16, New York, NY, USA. ACM.
Wang, Y., Tan, J., Yu,W., Zhang, L., Meng, X., and Li, X. (2013). Preemptive reducetask scheduling for fair and fast job completion. In Proceedings of the 10th International Conference on Autonomic Computing (ICAC 13), pages 279–289, San Jose, CA. USENIX.
Zaharia, M., Borthakur, D., Sen Sarma, J., Elmeleegy, K., Shenker, S., and Stoica, I. (2010). Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of the 5th European Conference on Computer Systems, EuroSys ’10, pages 265–278, New York, NY, USA. ACM.
Zaharia, M., Konwinski, A., Joseph, A. D., Katz, R., and Stoica, I. (2008). Improving mapreduce performance in heterogeneous environments. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI’08,n pages 29–42, Berkeley, CA, USA. USENIX Association.
Publicado
04/07/2016
Como Citar
SOUZA, Augusto; GARCIA, Islene.
A Preemptive Fair Scheduler Policy for Disco MapReduce Framework. In: WORKSHOP EM DESEMPENHO DE SISTEMAS COMPUTACIONAIS E DE COMUNICAÇÃO (WPERFORMANCE), 15. , 2016, Porto Alegre.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2016
.
p. 2758-2769.
ISSN 2595-6167.
DOI: https://doi.org/10.5753/wperformance.2016.9723.
