Shrinking Logs by Safely Discarding Commands
Resumo
Logs are crucial to the development of dependable distributed applications. By logging entries on a sequential global log, systems can synchronize updates over distributed replicas and provide a consistent state recovery in the presence of faults. However, logs account for a significant overhead on fault-tolerant applications' performance, and many studies present alternatives to alleviate servers from such costs. This paper proposes an approach to reduce log footprint by safely and efficiently discarding entries from logs. The expected benefits are twofold: minimize durability costs and speed up recovery. Besides shrinking logging information, the proposed technique splits the log into several files and incorporates strategies to reduce logging overhead, such as batching and parallel I/O. The proposed approach was compared to a standard logging scheme using realistic workloads. Results demonstrate that our logging approach is capable to generate compressed logs and reduce recovery time, imposing half the throughput overhead of a standard logging scheme.
Referências
Balakrishnan, M., Malkhi, D., Davis, J. D., Prabhakaran, V., Wei, M., and Wobber, T. (2013). Corfu: A distributed shared log. ACM Transactions on Computer Systems (TOCS), 31(4):1–24.
Bessani, A., Santos, M., Felix, J., Neves, N., and Correia, M. (2013). On the efficiency of durable state machine replication. In USENIX Annual Technical Conference, pages 169–180.
Budhiraja, N., Marzullo, K., Schneider, F. B., and Toueg, S. (1993). The primary-backup approach. Distributed systems, 2:199–216.
Celikel, Ö. and Ovatman, T. (2021). Distributed application checkpointing for replicated state machines. Scalable Computing: Practice and Experience, 22(1):67–79.
Chaczko, Z., Mahadevan, V., Aslanzadeh, S., and Mcdermid, C. (2011). Availability and load balancing in cloud computing. In ICCSM, Singapore.
Chandramouli, B., Prasaad, G., Kossmann, D., Levandoski, J., Hunter, J., and Barnett, M. (2018). Faster: A concurrent key-value store with in-place updates. In Proceedings of the 2018 International Conference on Management of Data, pages 275–290.
Clay, K. (2013). Amazon.com goes down, loses $66,240 per minute. [link].
Cooper, B. F., Silberstein, A., Tam, E., Ramakrishnan, R., and Sears, R. (2010). Benchmarking cloud serving systems with YCSB. In 1st ACM SoCC.
Facebook (2013). RocksDB. https://rocksdb.org/.
Junqueira, F. P., Reed, B. C., and Serafini, M. (2011). Zab: High-performance broadcast for primary-backup systems. In IEEE DSN, pages 245–256. IEEE.
Kreps, J. (2014). I Heart Logs: Event Data, Stream Processing, and Data Integration. O’Reilly Media, Inc.
Kreps, J., Narkhede, N., Rao, J., et al. (2011). Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB, volume 11, pages 1–7.
Lamport, L. (1978). Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7):558–565.
Liu, X., Iftikhar, N., and Xie, X. (2014). Survey of real-time processing systems for big data. In Proceedings of the 18th International Database Engineering & Applications Symposium, pages 356–361.
Mendizabal, O. M., Dotti, F. L., and Pedone, F. (2017). High performance recovery for In 2017 IEEE 37th International Conference on parallel state machine replication. Distributed Computing Systems (ICDCS), pages 34–44. IEEE.
Mohan, C., Haderle, D., Lindsay, B., Pirahesh, H., and Schwarz, P. (1992). Aries: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging. ACM Transactions on Database Systems (TODS), 17(1):94–162.
Narkhede, N., Shapira, G., and Palino, T. (2017). Kafka: the definitive guide: real-time data and stream processing at scale. O’Reilly Media, Inc.
Ongaro, D. and Ousterhout, J. (2014). In search of an understandable consensus algorithm. In USENIX Annual Technical Conference, pages 305–319.
White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad, S., Newbold, M., Hibler, M., Barb, C., and Joglekar, A. (2002). An integrated experimental environment for distributed systems and networks. ACM SIGOPS.
Xavier, L. G. C., Dotti, F. L., Meinhardt, C., and Mendizabal, O. M. (2020). Scalable and decoupled logging for state machine replication. In 38th Brazilian Symposium on Computer Networks and Distributed Systems (SBRC), pages 267–280. SBC.
Yao, C., Agrawal, D., Chen, G., Ooi, B. C., and Wu, S. (2016). Adaptive logging: Optimizing logging and recovery costs in distributed in-memory databases. In Proceedings of the 2016 International Conference on Management of Data, pages 1119–1134.
Yu, X., Zhu, S., Kaashoek, J., and Pavlo, A. (2016). Taurus: A Parallel Transaction Recovery Method Based on Fine-Granularity Dependency Tracking. CoRR.
Zhang, H., Chen, G., Ooi, B. C., Tan, K.-L., and Zhang, M. (2015). In-memory big data management and processing: A survey. IEEE Transactions on Knowledge and Data Engineering, 27(7):1920–1948.
Bessani, A., Santos, M., Felix, J., Neves, N., and Correia, M. (2013). On the efficiency of durable state machine replication. In USENIX Annual Technical Conference, pages 169–180.
Budhiraja, N., Marzullo, K., Schneider, F. B., and Toueg, S. (1993). The primary-backup approach. Distributed systems, 2:199–216.
Celikel, Ö. and Ovatman, T. (2021). Distributed application checkpointing for replicated state machines. Scalable Computing: Practice and Experience, 22(1):67–79.
Chaczko, Z., Mahadevan, V., Aslanzadeh, S., and Mcdermid, C. (2011). Availability and load balancing in cloud computing. In ICCSM, Singapore.
Chandramouli, B., Prasaad, G., Kossmann, D., Levandoski, J., Hunter, J., and Barnett, M. (2018). Faster: A concurrent key-value store with in-place updates. In Proceedings of the 2018 International Conference on Management of Data, pages 275–290.
Clay, K. (2013). Amazon.com goes down, loses $66,240 per minute. [link].
Cooper, B. F., Silberstein, A., Tam, E., Ramakrishnan, R., and Sears, R. (2010). Benchmarking cloud serving systems with YCSB. In 1st ACM SoCC.
Facebook (2013). RocksDB. https://rocksdb.org/.
Junqueira, F. P., Reed, B. C., and Serafini, M. (2011). Zab: High-performance broadcast for primary-backup systems. In IEEE DSN, pages 245–256. IEEE.
Kreps, J. (2014). I Heart Logs: Event Data, Stream Processing, and Data Integration. O’Reilly Media, Inc.
Kreps, J., Narkhede, N., Rao, J., et al. (2011). Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB, volume 11, pages 1–7.
Lamport, L. (1978). Time, clocks, and the ordering of events in a distributed system. Communications of the ACM, 21(7):558–565.
Liu, X., Iftikhar, N., and Xie, X. (2014). Survey of real-time processing systems for big data. In Proceedings of the 18th International Database Engineering & Applications Symposium, pages 356–361.
Mendizabal, O. M., Dotti, F. L., and Pedone, F. (2017). High performance recovery for In 2017 IEEE 37th International Conference on parallel state machine replication. Distributed Computing Systems (ICDCS), pages 34–44. IEEE.
Mohan, C., Haderle, D., Lindsay, B., Pirahesh, H., and Schwarz, P. (1992). Aries: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging. ACM Transactions on Database Systems (TODS), 17(1):94–162.
Narkhede, N., Shapira, G., and Palino, T. (2017). Kafka: the definitive guide: real-time data and stream processing at scale. O’Reilly Media, Inc.
Ongaro, D. and Ousterhout, J. (2014). In search of an understandable consensus algorithm. In USENIX Annual Technical Conference, pages 305–319.
White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad, S., Newbold, M., Hibler, M., Barb, C., and Joglekar, A. (2002). An integrated experimental environment for distributed systems and networks. ACM SIGOPS.
Xavier, L. G. C., Dotti, F. L., Meinhardt, C., and Mendizabal, O. M. (2020). Scalable and decoupled logging for state machine replication. In 38th Brazilian Symposium on Computer Networks and Distributed Systems (SBRC), pages 267–280. SBC.
Yao, C., Agrawal, D., Chen, G., Ooi, B. C., and Wu, S. (2016). Adaptive logging: Optimizing logging and recovery costs in distributed in-memory databases. In Proceedings of the 2016 International Conference on Management of Data, pages 1119–1134.
Yu, X., Zhu, S., Kaashoek, J., and Pavlo, A. (2016). Taurus: A Parallel Transaction Recovery Method Based on Fine-Granularity Dependency Tracking. CoRR.
Zhang, H., Chen, G., Ooi, B. C., Tan, K.-L., and Zhang, M. (2015). In-memory big data management and processing: A survey. IEEE Transactions on Knowledge and Data Engineering, 27(7):1920–1948.
Publicado
16/08/2021
Como Citar
XAVIER, Luiz Gustavo C.; DOTTI, Fernando Luís; MEINHARDT, Cristina; MENDIZABAL, Odorico M..
Shrinking Logs by Safely Discarding Commands. In: SIMPÓSIO BRASILEIRO DE REDES DE COMPUTADORES E SISTEMAS DISTRIBUÍDOS (SBRC), 39. , 2021, Uberlândia.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 588-601.
ISSN 2177-9384.
DOI: https://doi.org/10.5753/sbrc.2021.16749.