Using Machine Learning on Testing IoT Applications: a systematic mapping

• Lavínia Freitas UFC
• Valéria Lelli UFC

Resumo

Internet of Things (IoT) devices are increasingly present in people’s daily lives. Thus has increased research interest in investigating strategies that can ensure that these applications work as expected considering specific and vital characteristics of IoT, for example, security, performance and interoperability. In a testing point of view, there is a need to optimize and define an efficient strategy, from its planning to its execution. Considering all the steps that can be taken to test an IoT application, this process, if performed manually, can demand great effort and time. Machine learning (ML) algorithms have been applied in several areas of computing in order to optimize and automate processes that involve large volumes of data. In this paper, we present a systematic mapping resulting in 40 studies that highlights techniques or approaches that use machine learning algorithms for the most diverse goals within the IoT application testing process, such as the use of neural networks for predicting the cost of time in the preparation and execution of tests; identification of security attacks; and automatic generation of test cases from textual language. We also identified that the vast majority of testing techniques are focused on a specific IoT characteristic (e.g., security, performance), specially security, and apply the machine learning algorithm in two ways: directly in the algorithm, called predictive maintenance, or during the execution of planned tests, both of them bring difficulties related to extracting and defining data to train ML algorithms.

Palavras-chave: Software Testing, Machine Learning, Internet of Things, Systematic Mapping

Referências

Bernhard K Aichernig, Franz Pernkopf, Richard Schumi, and Andreas Wurm. 2019. Predicting and testing latencies with deep learning: An iot case study. In International Conference on Tests and Proofs. Springer, 93–111. https://doi.org/10.1007/978-3-030-31157-5_7

Irina Alam and Puneet Gupta. 2020. SAME-Infer: Software Assisted Memory Resilience for Efficient Inference at the Edge. In The International Symposium on Memory Systems. 10–22. https://doi.org/10.1145/3422575.3422774

Domenico Amalfitano, Nicola Amatucci, Vincenzo De Simone, Vincenzo Riccio, and Fasolino Anna Rita. 2017. Towards a Thing-In-the-Loop Approach for the Verification and Validation of IoT Systems. In Proceedings of the 1st ACM Workshop on the Internet of Safe Things (Delft, Netherlands) (SafeThings’17). Association for Computing Machinery, New York, NY, USA, 57–63. https://doi.org/10.1145/3137003.3137007

Rossana MC Andrade, Rainara M Carvalho, Italo Linhares de Araújo, Káthia M Oliveira, and Marcio EF Maia. 2017. What changes from ubiquitous computing to internet of things in interaction evaluation?. In International Conference on Distributed, Ambient, and Pervasive Interactions. Springer. https://doi.org/10.1007/978-3-319-58697-7_1

Simon D Duque Anton, Anna Pia Lohfink, Christoph Garth, and Hans Dieter Schotten. 2019. Security in process: Detecting attacks in industrial process data. In Proceedings of the Third Central European Cybersecurity Conference. 1–6. https://doi.org/10.1145/3360664.3360669

Francisco Arellano-Espitia, Miguel Delgado-Prieto, Victor Martínez-Viol, Ángel Fernández-Sobrino, and Roque Alfredo Osornio-Rios. 2021. Anomaly Detection in Electromechanical Systems by means of Deep-Autoencoder. In 2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA). IEEE, 01–06. https://doi.org/10.1109/ETFA45728.2021.9613529

Kevin ASHTON. 2009. A Coisa da Internet das Coisas. Artigo publicado em (2009).

Serkan Ayvaz and Koray Alpay. 2021. Predictive maintenance system for production lines in manufacturing: A machine learning approach using IoT data in real-time. Expert Systems with Applications 173 (2021), 114598. https://doi.org/10.1016/j.eswa.2021.114598

Antonia Bertolino, Antonello Calabró, Guglielmo De Angelis, Micael Gallego, Boni García, and Francisco Gortázar. 2018. When the testing gets tough, the tough get ElasTest. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion). IEEE, 17–20. https://doi.org/10.1145/3183440.3183497

Gerd Brewka. 1996. Artificial intelligence—a modern approach by Stuart Russell and Peter Norvig, Prentice Hall. Series in Artificial Intelligence, Englewood Cliffs, NJ. The Knowledge Engineering Review 11, 1 (1996), 78–79. https://doi.org/10.1017/S0269888900007724

Miroslav Bures, Tomas Cerny, and Bestoun S. Ahmed. 2019. Internet of Things: Current Challenges in the Quality Assurance and Testing Methods. In Information Science and Applications 2018, Kuinam J. Kim and Nakhoon Baek (Eds.). Springer Singapore, Singapore, 625–634. https://doi.org/10.1007/978-981-13-1056-0_61

Liana Carvalho, Valéria Lelli, and Rossana Andrade. 2022. Performance Testing Guide for IoT Applications. In Proceedings of the 24th International Conference on Enterprise Information Systems - Volume 1: ICEIS,. INSTICC, SciTePress, 667–678. https://doi.org/10.5220/0011090800003179

Mariela Cortés, Raphael Saraiva, Marcia Souza, Patricia Mello, and Pamella Soares. 2019. Adoption of software testing in internet of things: A systematic literature mapping. In Proceedings of the IV Brazilian Symposium on Systematic and Automated Software Testing. 3–11. https://doi.org/10.1145/3356317.3356326

Peter Danielis, Moritz Beckmann, and Jan Skodzik. 2020. An ISO-Compliant Test Procedure for Technical Risk Analyses of IoT Systems Based on STRIDE. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 499–504. https://doi.org/10.1109/COMPSAC48688.2020.0-203

Fabrizio De Vita, Dario Bruneo, and Sajal K Das. 2020. On the use of a full stack hardware/software infrastructure for sensor data fusion and fault prediction in industry 4.0. Pattern Recognition Letters 138 (2020), 30–37. https://doi.org/10.1016/j.patrec.2020.06.028

Vasily Desnitsky, Andrey Chechulin, and Igor Kotenko. 2022. Multi-Aspect Based Approach to Attack Detection in IoT Clouds. Sensors 22, 5 (2022), 1831. https://doi.org/10.3390/s22051831

Vinicius HS Durelli, Rafael S Durelli, Simone S Borges, Andre T Endo, Marcelo M Eler, Diego RC Dias, and Marcelo P Guimarães. 2019. Machine learning applied to software testing: A systematic mapping study. IEEE Transactions on Reliability 68, 3 (2019), 1189–1212. https://doi.org/10.1109/TR.2019.2892517

Thomas Fehlmann and Eberhard Kranich. 2017. Autonomous real-time software & systems testing. In Proceedings of the 27th International Workshop on Software Measurement and 12th International Conference on Software Process and Product Measurement. 54–63. https://doi.org/10.1145/3143434.3143444

Peter Flach. 2012. Machine learning: the art and science of algorithms that make sense of data. Cambridge University Press.

Anna Katrina Gomez and SimiKamini Bajaj. 2019. Challenges of testing complex Internet of Things (IoT) devices and systems. In 2019 11th International Conference on Knowledge and Systems Engineering (KSE). IEEE, 1–4. https://doi.org/10.1109/KSE.2019.8919324

VVF Grubisic, JPF Aguiar, and Z Simeu-Abazi. 2020. A Review on Intelligent Predictive Maintenance: Bibliometric analysis and new research directions. In 2020 International Conference on Control, Automation and Diagnosis (ICCAD). IEEE, 1–6. https://doi.org/10.1109/ICCAD49821.2020.9260504

Eman Hammad, Abhijit Kumar Nag, Anitha Chennamaneni, Mohsen Aghashahi, and Erdogan Dogdu. 2021. A Deep-Defense Approach for Next-Gen Cyber-Resilient Inter-Dependent Critical Infrastructure Systems. In 2021 Resilience Week (RWS). IEEE, 1–7. https://doi.org/10.1109/RWS52686.2021.9611790

Sohaib Hanif, Tuba Ilyas, and Muhammad Zeeshan. 2019. Intrusion detection in IoT using artificial neural networks on UNSW-15 dataset. In 2019 IEEE 16th International Conference on Smart Cities: Improving Quality of Life Using ICT & IoT and AI (HONET-ICT). IEEE, 152–156. https://doi.org/10.1109/HONET.2019.8908122

Kuo-Kai Hsieh, Li-C Wang, Wen Chen, and Jayanta Bhadra. 2017. Learning to produce direct tests for security verification using constrained process discovery. In 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC). IEEE, 1–6. https://doi.org/10.1145/3061639.3062271

Faisal Jamil, Dohyeun Kim, et al. 2021. An Ensemble of a Prediction and Learning Mechanism for Improving Accuracy of Anomaly Detection in Network Intrusion Environments. Sustainability 13, 18 (2021), 10057. https://doi.org/10.3390/su131810057

Kyeong-Joong Jeong, Jin-Duk Park, Kyusoon Hwang, Seong-Lyun Kim, and Won-Yong Shin. 2022. Two-Stage Deep Anomaly Detection With Heterogeneous Time Series Data. IEEE Access 10 (2022), 13704–13714. https://doi.org/10.1109/ACCESS.2022.3147188

Maxim Kalinin and Peter Zegzhda. 2020. AI-based Security for the Smart Networks. In 13th International Conference on Security of Information and Networks. 1–4. https://doi.org/10.1145/3433174.3433593

Kavin Kamaraj, Behnam Dezfouli, and Yuhong Liu. 2019. Edge mining on iot devices using anomaly detection. In 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE, 33–40. https://doi.org/10.1109/APSIPAASC47483.2019.9023207

Jalindar Karande and Sarang Joshi. 2021. DEDA: An algorithm for early detection of topology attacks in the internet of things. International Journal of Electrical and Computer Engineering 11, 2 (2021), 1761. https://doi.org/10.11591/ijece.v11i2.pp1761-1770

Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. (2007).

Vasiliy Krundyshev. 2020. Neural network approach to assessing cybersecurity risks in large-scale dynamic networks. In 13th International Conference on Security of Information and Networks. 1–8. https://doi.org/10.1145/3433174.3433603

Chu Luo, Jorge Goncalves, Eduardo Velloso, and Vassilis Kostakos. 2020. A survey of context simulation for testing mobile context-aware applications. ACM Computing Surveys (CSUR) 53, 1 (2020), 1–39. https://doi.org/10.1145/3372788

S Manimurugan. 2021. IoT-Fog-Cloud model for anomaly detection using improved Naïve Bayes and principal component analysis. Journal of Ambient Intelligence and Humanized Computing (2021), 1–10. https://doi.org/10.1007/s12652-020-02723-3

Noha Medhat, Sherin M Moussa, Nagwa Lotfy Badr, and Mohamed F Tolba. 2020. A Framework for Continuous Regression and Integration Testing in IoT Systems Based on Deep Learning and Search-Based Techniques. IEEE Access 8 (2020), 215716–215726. https://doi.org/10.1109/ACCESS.2020.3039931

Noha Medhat Mohamed, Sherin Moussa, Nagwa Badr, and Mohamed Tolba. 2021. Enhancing Test Cases Prioritization for Internet of Things based systems using Search-based Technique. International Journal of Intelligent Computing and Information Sciences 21, 1 (2021), 84–94. https://doi.org/10.21608/ijicis.2021.69462.1076

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Foundations of machine learning. MIT press.

W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. 2019. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences 116, 44 (2019), 22071–22080. https://doi.org/10.1073/pnas.1900654116

TD Ovasapyan, VD Danilov, and Dmitry A Moskvin. 2021. Application of synthetic data generation methods to the detection of network attacks on internet of things devices. Automatic Control and Computer Sciences 55, 8 (2021), 991–998. https://doi.org/10.3103/S0146411621080241

Keyur K Patel, Sunil M Patel, et al. 2016. Internet of things-IOT: definition, characteristics, architecture, enabling technologies, application & future challenges. International journal of engineering science and computing 6, 5 (2016). https://doi.org/10.4010/2016.1482

Ivan Porres, Tanwir Ahmad, Hergys Rexha, Sébastien Lafond, and Dragos Truscan. 2020. Automatic exploratory performance testing using a discriminator neural network. In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 105–113. https://doi.org/10.1109/ICSTW50294.2020.00030

P Shanmuga Prabha and S Magesh Kumar. 2022. A Novel Cyber-attack Leads Prediction System using Cascaded R2CNN Model. International Journal of Advanced Computer Science and Applications 13, 2 (2022). https://doi.org/10.14569/IJACSA.2022.0130260

Sumit Pundir, Mohammad S Obaidat, Mohammad Wazid, Ashok Kumar Das, Devesh Pratap Singh, and Joel JPC Rodrigues. 2021. MADP-IIME: malware attack detection protocol in IoT-enabled industrial multimedia environment using machine learning approach. Multimedia Systems (2021), 1–13. https://doi.org/10.1007/s00530-020-00743-9

Mamunur Rashid, Joarder Kamruzzaman, Tasadduq Imam, Santoso Wibowo, and Steven Gordon. 2022. A tree-based stacking ensemble technique with feature selection for network intrusion detection. Applied Intelligence (2022), 1–14. https://doi.org/10.1007/s10489-021-02968-1

Anas Saci, Arafat Al-Dweik, and Abdallah Shami. 2021. Autocorrelation in tegrated gaussian based anomaly detection using sensory data in industrial manufacturing. IEEE Sensors Journal 21, 7 (2021), 9231–9241. https://doi.org/10.1109/JSEN.2021.3053039

Ahmed Saeed, Ali Ahmadinia, Abbas Javed, and Hadi Larijani. 2016. Intelligent intrusion detection in low-power IoTs. ACM Transactions on Internet Technology (TOIT) 16, 4 (2016), 1–25. https://doi.org/10.1145/2990499

Safdar Aqeel Safdar, Tao Yue, and Shaukat Ali. 2021. Recommending Faulty Configurations for Interacting Systems Under Test Using Multi-objective Search. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 4 (2021), 1–36. https://doi.org/10.1145/3464939

Firooz B Saghezchi, Georgios Mantas, Manuel A Violas, A Manuel de Oliveira Duarte, and Jonathan Rodriguez. 2022. Machine learning for DDoS attack detection in industry 4.0 CPPSs. Electronics 11, 4 (2022), 602. https://doi.org/10.3390/electronics11040602

Iqbal H Sarker. 2021. Cyberlearning: effectiveness analysis of machine learning security modeling to detect cyber-anomalies and multi-attacks. Internet of Things 14 (2021), 100393. https://doi.org/10.1016/j.iot.2021.100393

Iqbal H Sarker, Yoosef B Abushark, Fawaz Alsolami, and Asif Irshad Khan. 2020. Intrudtree: a machine learning based cyber security intrusion detection model. Symmetry 12, 5 (2020), 754. https://doi.org/10.3390/sym12050754

Iqbal H Sarker, Asif Irshad Khan, Yoosef B Abushark, and Fawaz Alsolami. 2022. Internet of things (iot) security intelligence: a comprehensive overview, machine learning solutions and research directions. Mobile Networks and Applications (2022), 1–17. https://doi.org/10.1007/s11036-022-01937-3

V Sathyavathy and D Shanmuga Priyaa. 2018. Software Testing Techniques with Artificial Intelligence in IoT Applications. International Journal of Recent Technology and Engineering (IJRTE) 7 (2018).

Le-Tian Sha, Fu Xiao, Hai-Ping Huang, Yu Chen, and Ru-Chuan Wang. 2019. Catching Escapers: A Detection Method for Advanced Persistent Escapers in Industry Internet of Things Based on Identity-based Broadcast Encryption (IBBE). ACM Transactions on Embedded Computing Systems (TECS) 18, 3 (2019), 1–25. https://doi.org/10.1145/3319615

Vishal Sharma, Ilsun You, Kangbin Yim, Ray Chen, and Jin-Hee Cho. 2019. BRIoT: Behavior rule specification-based misbehavior detection for IoT-embedded cyberphysical systems. IEEE Access 7 (2019), 118556–118580. https://doi.org/10.1109/ACCESS.2019.2917135

Dr Ovidiu Vermesan SINTEF and Dr Norway. 2014. Peter FriessEU, Belgium,“Internet of Things–From Research and Innovation to Market Deployment”.

Ian Sommerville. 2010. Software Engineering (9 ed.). Addison-Wesley, Harlow, England.

Ovidiu Vermesan and Peter Friess. 2013. Internet of things: converging technologies for smart environments and integrated ecosystems. River publishers.

Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering. 1–10. https://doi.org/10.1145/2601248.2601268

Tianwei Xing, Luis Garcia, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava. 2021. DeepSQA: Understanding Sensor Data via Question Answering. In Proceedings of the International Conference on Internet-of-Things Design and Implementation. 106–118. https://doi.org/10.1145/3450268.3453529

Wenjin Yu, Tharam Dillon, Fahed Mostafa, Wenny Rahayu, and Yuehua Liu. 2019. A global manufacturing big data ecosystem for fault detection in predictive maintenance. IEEE Transactions on Industrial Informatics 16, 1 (2019), 183–192. https://doi.org/10.1109/TII.2019.2915846

Wenjin Yu, Yuehua Liu, Tharam Dillon, Wenny Rahayu, and Fahed Mostafa. 2021. An integrated framework for health state monitoring in a smart factory employing IoT and big data techniques. IEEE Internet of Things Journal (2021). https://doi.org/10.1109/JIOT.2021.3096637

Ying Zhao, Lei Wang, Shijie Li, Fangfang Zhou, Xiaoru Lin, Qiang Lu, and Lei Ren. 2019. A visual analysis approach for understanding durability test data of automotive products. ACM Transactions on Intelligent Systems and Technology (TIST) 10, 6 (2019), 1–23. https://doi.org/10.1145/3345640