Defigium: A Modular Framework for Trace-Based Workload Generation
Resumo
Performance testing for distributed systems depends heavily on the accurate reproduction of realistic workloads. Across domains such as scientific computing, cloud computing, and enterprise services, request patterns vary widely in terms of arrival rates, resource access popularity, and other application-specific characteristics. Furthermore, user behavior may change over time, as some applications exhibit pronounced seasonality, such as workload spikes during Black Friday or major sports events. Existing benchmarking standards frequently fail to capture this temporal complexity and the nuanced access patterns found in production environments. In this paper, we present Defigium, a modular framework designed to generate customized benchmark workloads by analyzing real execution traces. The framework employs a three-stage pipeline consisting of trace-based parsing, workload model generation, and load execution. To demonstrate Defigium’s modularity and utility, we implemented parsing modules for Redis and HTTP to cover key-value stores and web application examples. We further developed two distinct workload model generation modules: a standard replay and a compacted version utilizing heatmaps to capture the frequency of operations and inter-arrival times. Our evaluation shows that Defigium provides extensibility and reuse while effectively characterizing dynamic production environments.
Referências
Cao, Z., Dong, S., Vemuri, S., and Du, D. H. (2020). Characterizing, modeling, and benchmarking RocksDB Key-Value workloads at Facebook. In 18th USENIX Conference on File and Storage Technologies (FAST 20), pages 209–223.
Cohen, J. (2013). The test of goodness of fit and contingency tables. In Statistical Power Analysis for the Behavioral Sciences. Taylor & Francis.
Cooper, B. F., Silberstein, A., Tam, E., Ramakrishnan, R., and Sears, R. (2010). Benchmarking cloud serving systems with ycsb. In Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC ’10, page 143–154, New York, NY, USA. Association for Computing Machinery.
Delimitrou, C. and Kozyrakis, C. (2011). Accurate modeling and generation of storage i/o for datacenter workloads. ACM Transactions on Storage, 7(3):8:1–8:23.
Du, M., Li, F., Zheng, G., and Srikumar, V. (2017). Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 1285–1298.
Ghandeharizadeh, S. and Huang, H. (2018). Hoagie: A database and workload generator using published specifications. In 2018 IEEE International Conference on Big Data (Big Data), pages 3847–3852.
Gmach, D., Rolia, J., Cherkasova, L., and Kemper, A. (2007). Workload analysis and demand prediction of enterprise data center applications. In Proceedings of the 2007 IEEE International Symposium on Workload Characterization, IISWC, pages 171–180.
Khan, A., Yan, X., Tao, S., and Anerousis, N. (2012). Workload characterization and prediction in the cloud: A multiple time series approach. In 2012 IEEE Network Operations and Management Symposium, pages 1287–1294.
Kim, K., Lee, C., Jung, J. H., and Ro, W. W. (2014). Workload synthesis: Generating benchmark workloads from statistical execution profile. In 2014 IEEE International Symposium on Workload Characterization (IISWC), pages 120–129.
Lin, M., Lucas, H., and Shmueli, G. (2013). Too big to fail: Large samples and the p-value problem. Information Systems Research, 24:906–917.
Lorido-Botran, T., Miguel-Alonso, J., and Lozano, J. A. (2014). A review of auto-scaling techniques for elastic applications in cloud environments. Journal of grid computing, 12(4):559–592.
Maalej, W., Nayebi, M., Johann, T., and Ruhe, G. (2015). Toward data-driven requirements engineering. IEEE software, 33(1):48–54.
Mendizabal, O. M., Spier, M., and Saad, R. (2012). Log-based approach for performance requirements elicitation and prioritization. In 2012 20th IEEE International Requirements Engineering Conference (RE), pages 297–302. IEEE.
Sfakianakis, Y., Kanellou, E., Marazakis, M., and Bilas, A. (2021). Trace-based workload generation and execution. In International Conference on Parallel and Distributed Computing (Euro-Par).
Sladojević, V., Frischbier, S., Echler, A., Paic, M., and Margara, A. (2022). Deriving a realistic workload model to simulate high-volume financial data feeds for performance benchmarking. In Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems, DEBS ’22, page 126–131, New York, NY, USA. Association for Computing Machinery.
Talluri, S., Łuszczak, A., Abad, C. L., and Iosup, A. (2019). Characterization of a big data storage workload in the cloud. In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering, ICPE ’19, page 33–44, New York, NY, USA. Association for Computing Machinery.
TPC (1988). TPC-Homepage.
White, B., Lepreau, J., Stoller, L., Ricci, R., Guruprasad, S., Newbold, M., Hibler, M., Barb, C., and Joglekar, A. (2002). An integrated experimental environment for dis tributed systems and networks. ACM SIGOPS Operating Systems Review, 36(SI):255–270.
Yang, J., Yue, Y., and Rashmi, K. (2021). A large-scale analysis of hundreds of in-memory key-value cache clusters at twitter. ACM Transactions on Storage (TOS), 17(3).
Zaker, F. (2019). Online Shopping Store - Web Server Logs.
Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., and Ding, D. (2018). Fault analysis and debugging of microservice systems: Industrial survey, benchmark system, and empirical study. IEEE Transactions on Software Engineering, 47(2):243–260.
