Evaluating the Effectiveness of MQTT Fuzzing Tools Using Mutation Testing in IoT Systems
Resumo
Internet of Things (IoT) systems are widely used in residential and industrial domains, with MQTT as a common communication protocol. However, testing MQTT-based systems remains challenging due to their event-driven behavior, preconfigured topics, asynchronous processing, and frequent silent failures. Although several MQTT fuzzing tools have been proposed, there is limited empirical evidence comparing their effectiveness in realistic IoT platforms using adequacy criteria beyond crash detection. This paper presents a controlled experiment evaluating MQTT fuzzing tools using Home Assistant, an open-source home automation platform deployed in a reproducible Docker-based environment. Effectiveness is assessed through mutation testing, with artificial faults injected into the source code and a stratified oracle detecting crashes, internal exceptions, incorrect entity states, and silent behavioral divergences. The evaluation uses mutation score as the main metric, complemented by an MQTT-adjusted mutation score, killed-mutant overlap, and post-hoc analysis of surviving mutants. The results provide empirical evidence on the ability of each fuzzer to expose application-level faults and highlight limitations related to reachability, configuration-dependent execution paths, and oracle observability in MQTT-based IoT systems.
Referências
Tanweer Alam. 2018. A Reliable Communication Framework and Its Use in Internet of Things (IoT). International Journal of Scientific Research in Computer Science, Engineering and Information Technology 3, 5 (2018).
Luis Gustavo Araujo Rodriguez and Daniel Macêdo Batista. 2020. Program-Aware Fuzzing for MQTT Applications. In 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’20). ACM, 582–586.
Luis Gustavo Araujo Rodriguez and Daniel Macêdo Batista. 2021. Towards Improving Fuzzer Efficiency for the MQTT Protocol. In 2021 IEEE Symposium on Computers and Communications (ISCC). IEEE, 1–7.
Philippe Biondi and The Scapy Community. 2023. Scapy: A powerful interactive packet manipulation program. [link].
Samin Y. Chowdhury, Ruimin Sun, and Brandon Dudley. 2025. A Survey for MQTT Fuzzing. In Proceedings of the 2025Workshop on Re-design Industrial Control Systems with Security (RICSS ’25). ACM.
Maialen Eceiza, Jose Luis Flores, and Mikel Iturbe. 2021. Fuzzing the Internet of Things: A Review on the Techniques and Challenges for Efficient Vulnerability Discovery in Embedded Systems. IEEE Internet of Things Journal 8, 13 (2021), 10390–10411.
Bruno Garcia and Simone Souza. 2025. An empirical evaluation of fuzz targets using mutation testing. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 75–83.
Santiago Hernández Ramos, M. Teresa Villalba, and Raquel Lacuesta. 2018. MQTT Security: A Novel Fuzzing Approach. Wireless Communications and Mobile Computing 2018 (2018).
Jaekwon Lee, Enrico Viganò, Fabrizio Pastore, and Lionel Briand. 2024. MOTIF: A tool for Mutation Testing with Fuzzing. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). 451–453.
Xinpeng Liu, Qinying Wang, Peiyu Liu, Wenhai Wang, and Shouling Ji. 2025. MQueez: Specification-Driven Fuzzing for MQTT Broker (Registered Report). In Proceedings of the 34th ACM SIGSOFT International Symposium on Software Testing and Analysis Companion (ISSTA Companion ’25). ACM.
Valentin J. M. Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2021. The Art, Science, and Engineering of Fuzzing: A Survey. IEEE Transactions on Software Engineering 47, 11 (2021), 2312–2331. DOI: 10.1109/TSE.2019.2946563
Jean Baptiste Minani, Fatima Sabir, Naouel Moha, and Yann-Gaël Gueéneuc. 2024. A Systematic Review of IoT Systems Testing: Objectives, Approaches, Tools, and Challenges. IEEE Transactions on Software Engineering (2024).
Biswajeeban Mishra and Attila Kertész. 2020. The Use of MQTT in M2M and IoT Systems: A Survey. IEEE Access (2020). DOI: 10.1109/ACCESS.2020.3035849
Bryan Pearson, Yue Zhang, Cliff Zou, and Xinwen Fu. 2022. FUME: Fuzzing Message Queuing Telemetry Transport Brokers. In IEEE INFOCOM 2022 - IEEE Conference on Computer Communications. IEEE, 1699–1708.
Jesse Pereyda. 2024. BooFuzz: Network Protocol Fuzzing for Humans. [link].
Pallavi Sethi and Smruti Sarangi. 2017. Internet of Things: Architectures, Protocols, and Applications. J. of Electrical and Computer Engineering (2017), 1–25.
Xiangpu Song, Jianliang Wu, Yingpei Zeng, Hao Pan, Chaoshun Zuo, Qingchuan Zhao, and Shanqing Guo. 2025. MBFuzzer: A Multi-Party Protocol Fuzzer for MQTT Brokers. In 34th USENIX Security Symposium. USENIX Association.
Vasudev Vikram, Isabella Laybourn, Ao Li, Nicole Nair, Kelton OBrien, Rafaello Sanna, and Rohan Padhye. 2023. Guiding Greybox Fuzzing with Mutation Testing. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (Seattle, WA, USA) (ISSTA 2023). Association for Computing Machinery, New York, NY, USA, 929–941. DOI: 10.1145/3597926.3598107
