A Q-Learning approach for dynamic scheduling of TSCH communication

• Victor S. Cardel UFBA
• Paulo H. L. Rettore Fraunhofer FKIE
• Bruno P. Santos UFBA

DOI: https://doi.org/10.5753/sbrc.2024.1275

Abstract

A 6TiSCH network provides IPv6 connectivity through links IEEE 802.15.4 governed by Time Slotted Channel Hopping (TSCH). Essentially, TSCH intends to provide low energy consumption and high reliability by scheduling communication resources and the channel hopping mechanism. However, 6TiSCH does not define the policies to build and maintain the communication schedule. This work proposes a new scheduling function that utilizes Q-Learning, considering the network traffic variation, the energy consumption, and the message queue size to be sent by the device. We compared the proposed method with Minimal Scheduling Function (MSF), the de facto scheduler used in the literature. The experiments show that the proposed method reduces the communication latency while maintaining high reliability and low energy consumption and join times. This indicates the potential of the approach.

References

Bekar, T., Görmüş, S., Aydın, B., and Aydın, H. (2023). Q-Learning Algorithm Inspired Objective Function Optimization For IETF 6TiSCH Networks. In SmartNets. IEEE.

Brandt, A., Designs, S., Hui, J., Kelsey, R., Levis, P., Pister, K., Struik, R., Vasseur, J., and Alexander, R. (2012). Internet engineering task force (ietf) t. winter, ed. request for comments: 6550 category: Standards track p. thubert, ed.

Clifton, J. and Laber, E. (2020). Q-learning: Theory and applications. Annual Review of Statistics and Its Application, 7:279–301.

Domingo-Prieto, M., Chang, T., Vilajosana, X., and Watteyne, T. (2016). Distributed pid-based scheduling for 6tisch networks. IEEE Communications Letters, 20(5):1006–1009.

Duquennoy, S., Al Nahas, B., Landsiedel, O., and Watteyne, T. (2015). Orchestra: Robust mesh networks through autonomously scheduled TSCH. In Proceedings of the 13th ACM conference on embedded networked sensor systems.

Duquennoy, S., Elsts, A., Al Nahas, B., and Oikonomo, G. (2017). Tsch and 6tisch for contiki: Challenges, design and evaluation. In 13th International Conference on Distributed Computing in Sensor Systems (DCOSS). IEEE.

Fawwaz, D. Z. and Chung, S.-H. (2023). Adaptive Trickle Timer for Efficient 6TiSCH Network Formation using Q-Learning. IEEE Access.

Ha, Y. and Chung, S.-H. (2022). Traffic-Aware 6TiSCH Routing Method for IIoT Wireless Networks. IEEE Internet of Things Journal.

Hamza, T. and Kaddoum, G. (2019). Enhanced minimal scheduling function for IEEE 802.15. 4e TSCH networks. In WCNC. IEEE.

Hauweele, D., Koutsiamanis, R.-A., Quoitin, B., and Papadopoulos, G. Z. (2020). Pushing 6TiSCH minimal scheduling function (MSF) to the limits. In 2020 IEEE Symposium on Computers and Communications (ISCC), pages 1–7. IEEE.

Hazra, A., Adhikari, M., Amgoth, T., and Srirama, S. N. (2021). A comprehensive survey on interoperability for IIoT: Taxonomy, standards, and future directions. ACM Computing Surveys (CSUR), 55(1):1–35.

Kalita, A. and Khatua, M. (2022). 6tisch–ipv6 enabled open stack iot network formation: A review. ACM Transactions on Internet of Things, 3(3):1–36.

Kherbache, M., Sobirov, O., Maimour, M., Rondeau, E., and Benyahia, A. (2023). Decentralized TSCH scheduling protocols and heterogeneous traffic: Overview and performance evaluation. Internet of Things, page 100696.

Le, H., John, M., and Pister, K. (2009). Energy-Aware Routing in Wireless Sensor Networks with Adaptive Energy-Slope Control; EE290Q-2 Spring. IEEE: Piscataway, NJ, USA.

Municio, E., Daneels, G., Vučinić, M., Latré, S., Famaey, J., Tanaka, Y., Brun, K., Muraoka, K., Vilajosana, X., and Watteyne, T. (2019). Simulating 6TiSCH networks. Transactions on Emerging Telecommunications Technologies.

Nguyen-Duy, H., Ngo-Quynh, T., Kojima, F., Pham-Van, T., Nguyen-Duc, T., and Luongoudon, S. (2019). RL-TSCH: A Reinforcement Learning Algorithm for Radio Scheduling in TSCH 802.15.4e. In ICTC, pages 227–231. IEEE.

Palattella, M. R., Accettura, N., Dohler, M., Grieco, L. A., and Boggia, G. (2012). Traffic aware scheduling algorithm for reliable low-power multi-hop IEEE 802.15. 4e networks. In PIMRC, pages 327–332. IEEE.

Peter, O., Pradhan, A., and Mbohwa, C. (2023). Industrial internet of things (iiot): opportunities, challenges, and requirements in manufacturing businesses in emerging economies. Procedia Computer Science, 217:856–865.

Pratama, Y. H. and Chung, S. (2022). RL-SF: Reinforcement Learning based Scheduling Function for Distributed TSCH Networks. In IEEE ICEIEC, pages 5–8.

Santos, B. P., Rettore, P. H., Vieira, L. F. M., and Loureiro, A. A. F. (2019). Dribble: A learn-based timer scheme selector for mobility management in IoT. In 2019 IEEE Wireless Communications and Networking Conference (WCNC).

Santos, B. P., Silva, L. A., Celes, C., Borges, J. B., Neto, B. S. P., Vieira, M. A. M., Vieira, L. F. M., Goussevskaia, O. N., and Loureiro, A. (2016). Internet das coisas: da teoria à prática. Minicursos SBRC, 31:16.

Thubert, P. (2021). RFC 9030: An Architecture for IPv6 over the Time-Slotted Channel Hopping Mode of IEEE 802.15. 4 (6TiSCH).

Ullah, Z., Al-Turjman, F., Mostarda, L., and Gagliardi, R. (2020). Applications of artificial intelligence and machine learning in smart cities. Computer Communications, 154:313–323.

Vilajosana, X., Wang, Q., Chraim, F., Watteyne, T., Chang, T., and Pister, K. S. (2013). A realistic energy consumption model for TSCH networks. IEEE Sensors Journal, 14(2):482–489.

Vilajosana, X., Watteyne, T., Chang, T., Vučinić, M., Duquennoy, S., and Thubert, P. (2019). Ietf 6tisch: A tutorial. IEEE Communications Surveys & Tutorials, 22(1):595–615.

Wang, Q., Vilajosana, X., and Watteyne, T. (2018). 6TiSCH Operation Sublayer (6top) Protocol (6P)-RFC8480. Internet Engineering Task Force RFC series.

Wiering, M. A. and Van Otterlo, M. (2012). Reinforcement learning. Adaptation, learning, and optimization, 12(3):729.