Uma abordagem Q-Learning para escalonamento dinâmico de comunicação do TSCH
Resumo
Uma rede mesh 6TiSCH provê conectividade IPv6 usando enlaces IEEE 802.15.4 governados pelo Time Slotted Channel Hopping (TSCH). Essencialmente, o TSCH promete baixo consumo de energia e alta confiabilidade através do escalonamento de tempo e salto de canais de comunicação, respectivamente. Entretanto, o 6TiSCH não define as políticas para construir e manter o cronograma de comunicação. Este trabalho propõe uma nova função de escalonamento de comunicação que utiliza Q-Learning, que leva em consideração a variação no tráfego da rede, o consumo de energia e o tamanho da fila de mensagens a serem enviadas pelo dispositivo. Comparamos a abordagem proposta com Minimal Scheduling Function (MSF), o escalonador de facto usada na literatura. Os experimentos mostram que a abordagem proposta reduz a latência da comunicação, enquanto mantém a confiabilidade alta, o consumo de energia e tempo de junção da rede baixos, mostrando que a abordagem é promissora.Referências
Bekar, T., Görmüş, S., Aydın, B., and Aydın, H. (2023). Q-Learning Algorithm Inspired Objective Function Optimization For IETF 6TiSCH Networks. In SmartNets. IEEE.
Brandt, A., Designs, S., Hui, J., Kelsey, R., Levis, P., Pister, K., Struik, R., Vasseur, J., and Alexander, R. (2012). Internet engineering task force (ietf) t. winter, ed. request for comments: 6550 category: Standards track p. thubert, ed.
Clifton, J. and Laber, E. (2020). Q-learning: Theory and applications. Annual Review of Statistics and Its Application, 7:279–301.
Domingo-Prieto, M., Chang, T., Vilajosana, X., and Watteyne, T. (2016). Distributed pid-based scheduling for 6tisch networks. IEEE Communications Letters, 20(5):1006–1009.
Duquennoy, S., Al Nahas, B., Landsiedel, O., and Watteyne, T. (2015). Orchestra: Robust mesh networks through autonomously scheduled TSCH. In Proceedings of the 13th ACM conference on embedded networked sensor systems.
Duquennoy, S., Elsts, A., Al Nahas, B., and Oikonomo, G. (2017). Tsch and 6tisch for contiki: Challenges, design and evaluation. In 13th International Conference on Distributed Computing in Sensor Systems (DCOSS). IEEE.
Fawwaz, D. Z. and Chung, S.-H. (2023). Adaptive Trickle Timer for Efficient 6TiSCH Network Formation using Q-Learning. IEEE Access.
Ha, Y. and Chung, S.-H. (2022). Traffic-Aware 6TiSCH Routing Method for IIoT Wireless Networks. IEEE Internet of Things Journal.
Hamza, T. and Kaddoum, G. (2019). Enhanced minimal scheduling function for IEEE 802.15. 4e TSCH networks. In WCNC. IEEE.
Hauweele, D., Koutsiamanis, R.-A., Quoitin, B., and Papadopoulos, G. Z. (2020). Pushing 6TiSCH minimal scheduling function (MSF) to the limits. In 2020 IEEE Symposium on Computers and Communications (ISCC), pages 1–7. IEEE.
Hazra, A., Adhikari, M., Amgoth, T., and Srirama, S. N. (2021). A comprehensive survey on interoperability for IIoT: Taxonomy, standards, and future directions. ACM Computing Surveys (CSUR), 55(1):1–35.
Kalita, A. and Khatua, M. (2022). 6tisch–ipv6 enabled open stack iot network formation: A review. ACM Transactions on Internet of Things, 3(3):1–36.
Kherbache, M., Sobirov, O., Maimour, M., Rondeau, E., and Benyahia, A. (2023). Decentralized TSCH scheduling protocols and heterogeneous traffic: Overview and performance evaluation. Internet of Things, page 100696.
Le, H., John, M., and Pister, K. (2009). Energy-Aware Routing in Wireless Sensor Networks with Adaptive Energy-Slope Control; EE290Q-2 Spring. IEEE: Piscataway, NJ, USA.
Municio, E., Daneels, G., Vučinić, M., Latré, S., Famaey, J., Tanaka, Y., Brun, K., Muraoka, K., Vilajosana, X., and Watteyne, T. (2019). Simulating 6TiSCH networks. Transactions on Emerging Telecommunications Technologies.
Nguyen-Duy, H., Ngo-Quynh, T., Kojima, F., Pham-Van, T., Nguyen-Duc, T., and Luongoudon, S. (2019). RL-TSCH: A Reinforcement Learning Algorithm for Radio Scheduling in TSCH 802.15.4e. In ICTC, pages 227–231. IEEE.
Palattella, M. R., Accettura, N., Dohler, M., Grieco, L. A., and Boggia, G. (2012). Traffic aware scheduling algorithm for reliable low-power multi-hop IEEE 802.15. 4e networks. In PIMRC, pages 327–332. IEEE.
Peter, O., Pradhan, A., and Mbohwa, C. (2023). Industrial internet of things (iiot): opportunities, challenges, and requirements in manufacturing businesses in emerging economies. Procedia Computer Science, 217:856–865.
Pratama, Y. H. and Chung, S. (2022). RL-SF: Reinforcement Learning based Scheduling Function for Distributed TSCH Networks. In IEEE ICEIEC, pages 5–8.
Santos, B. P., Rettore, P. H., Vieira, L. F. M., and Loureiro, A. A. F. (2019). Dribble: A learn-based timer scheme selector for mobility management in IoT. In 2019 IEEE Wireless Communications and Networking Conference (WCNC).
Santos, B. P., Silva, L. A., Celes, C., Borges, J. B., Neto, B. S. P., Vieira, M. A. M., Vieira, L. F. M., Goussevskaia, O. N., and Loureiro, A. (2016). Internet das coisas: da teoria à prática. Minicursos SBRC, 31:16.
Thubert, P. (2021). RFC 9030: An Architecture for IPv6 over the Time-Slotted Channel Hopping Mode of IEEE 802.15. 4 (6TiSCH).
Ullah, Z., Al-Turjman, F., Mostarda, L., and Gagliardi, R. (2020). Applications of artificial intelligence and machine learning in smart cities. Computer Communications, 154:313–323.
Vilajosana, X., Wang, Q., Chraim, F., Watteyne, T., Chang, T., and Pister, K. S. (2013). A realistic energy consumption model for TSCH networks. IEEE Sensors Journal, 14(2):482–489.
Vilajosana, X., Watteyne, T., Chang, T., Vučinić, M., Duquennoy, S., and Thubert, P. (2019). Ietf 6tisch: A tutorial. IEEE Communications Surveys & Tutorials, 22(1):595–615.
Wang, Q., Vilajosana, X., and Watteyne, T. (2018). 6TiSCH Operation Sublayer (6top) Protocol (6P)-RFC8480. Internet Engineering Task Force RFC series.
Wiering, M. A. and Van Otterlo, M. (2012). Reinforcement learning. Adaptation, learning, and optimization, 12(3):729.
Brandt, A., Designs, S., Hui, J., Kelsey, R., Levis, P., Pister, K., Struik, R., Vasseur, J., and Alexander, R. (2012). Internet engineering task force (ietf) t. winter, ed. request for comments: 6550 category: Standards track p. thubert, ed.
Clifton, J. and Laber, E. (2020). Q-learning: Theory and applications. Annual Review of Statistics and Its Application, 7:279–301.
Domingo-Prieto, M., Chang, T., Vilajosana, X., and Watteyne, T. (2016). Distributed pid-based scheduling for 6tisch networks. IEEE Communications Letters, 20(5):1006–1009.
Duquennoy, S., Al Nahas, B., Landsiedel, O., and Watteyne, T. (2015). Orchestra: Robust mesh networks through autonomously scheduled TSCH. In Proceedings of the 13th ACM conference on embedded networked sensor systems.
Duquennoy, S., Elsts, A., Al Nahas, B., and Oikonomo, G. (2017). Tsch and 6tisch for contiki: Challenges, design and evaluation. In 13th International Conference on Distributed Computing in Sensor Systems (DCOSS). IEEE.
Fawwaz, D. Z. and Chung, S.-H. (2023). Adaptive Trickle Timer for Efficient 6TiSCH Network Formation using Q-Learning. IEEE Access.
Ha, Y. and Chung, S.-H. (2022). Traffic-Aware 6TiSCH Routing Method for IIoT Wireless Networks. IEEE Internet of Things Journal.
Hamza, T. and Kaddoum, G. (2019). Enhanced minimal scheduling function for IEEE 802.15. 4e TSCH networks. In WCNC. IEEE.
Hauweele, D., Koutsiamanis, R.-A., Quoitin, B., and Papadopoulos, G. Z. (2020). Pushing 6TiSCH minimal scheduling function (MSF) to the limits. In 2020 IEEE Symposium on Computers and Communications (ISCC), pages 1–7. IEEE.
Hazra, A., Adhikari, M., Amgoth, T., and Srirama, S. N. (2021). A comprehensive survey on interoperability for IIoT: Taxonomy, standards, and future directions. ACM Computing Surveys (CSUR), 55(1):1–35.
Kalita, A. and Khatua, M. (2022). 6tisch–ipv6 enabled open stack iot network formation: A review. ACM Transactions on Internet of Things, 3(3):1–36.
Kherbache, M., Sobirov, O., Maimour, M., Rondeau, E., and Benyahia, A. (2023). Decentralized TSCH scheduling protocols and heterogeneous traffic: Overview and performance evaluation. Internet of Things, page 100696.
Le, H., John, M., and Pister, K. (2009). Energy-Aware Routing in Wireless Sensor Networks with Adaptive Energy-Slope Control; EE290Q-2 Spring. IEEE: Piscataway, NJ, USA.
Municio, E., Daneels, G., Vučinić, M., Latré, S., Famaey, J., Tanaka, Y., Brun, K., Muraoka, K., Vilajosana, X., and Watteyne, T. (2019). Simulating 6TiSCH networks. Transactions on Emerging Telecommunications Technologies.
Nguyen-Duy, H., Ngo-Quynh, T., Kojima, F., Pham-Van, T., Nguyen-Duc, T., and Luongoudon, S. (2019). RL-TSCH: A Reinforcement Learning Algorithm for Radio Scheduling in TSCH 802.15.4e. In ICTC, pages 227–231. IEEE.
Palattella, M. R., Accettura, N., Dohler, M., Grieco, L. A., and Boggia, G. (2012). Traffic aware scheduling algorithm for reliable low-power multi-hop IEEE 802.15. 4e networks. In PIMRC, pages 327–332. IEEE.
Peter, O., Pradhan, A., and Mbohwa, C. (2023). Industrial internet of things (iiot): opportunities, challenges, and requirements in manufacturing businesses in emerging economies. Procedia Computer Science, 217:856–865.
Pratama, Y. H. and Chung, S. (2022). RL-SF: Reinforcement Learning based Scheduling Function for Distributed TSCH Networks. In IEEE ICEIEC, pages 5–8.
Santos, B. P., Rettore, P. H., Vieira, L. F. M., and Loureiro, A. A. F. (2019). Dribble: A learn-based timer scheme selector for mobility management in IoT. In 2019 IEEE Wireless Communications and Networking Conference (WCNC).
Santos, B. P., Silva, L. A., Celes, C., Borges, J. B., Neto, B. S. P., Vieira, M. A. M., Vieira, L. F. M., Goussevskaia, O. N., and Loureiro, A. (2016). Internet das coisas: da teoria à prática. Minicursos SBRC, 31:16.
Thubert, P. (2021). RFC 9030: An Architecture for IPv6 over the Time-Slotted Channel Hopping Mode of IEEE 802.15. 4 (6TiSCH).
Ullah, Z., Al-Turjman, F., Mostarda, L., and Gagliardi, R. (2020). Applications of artificial intelligence and machine learning in smart cities. Computer Communications, 154:313–323.
Vilajosana, X., Wang, Q., Chraim, F., Watteyne, T., Chang, T., and Pister, K. S. (2013). A realistic energy consumption model for TSCH networks. IEEE Sensors Journal, 14(2):482–489.
Vilajosana, X., Watteyne, T., Chang, T., Vučinić, M., Duquennoy, S., and Thubert, P. (2019). Ietf 6tisch: A tutorial. IEEE Communications Surveys & Tutorials, 22(1):595–615.
Wang, Q., Vilajosana, X., and Watteyne, T. (2018). 6TiSCH Operation Sublayer (6top) Protocol (6P)-RFC8480. Internet Engineering Task Force RFC series.
Wiering, M. A. and Van Otterlo, M. (2012). Reinforcement learning. Adaptation, learning, and optimization, 12(3):729.
Publicado
20/05/2024
Como Citar
CARDEL, Victor S.; RETTORE, Paulo H. L.; SANTOS, Bruno P..
Uma abordagem Q-Learning para escalonamento dinâmico de comunicação do TSCH. In: SIMPÓSIO BRASILEIRO DE REDES DE COMPUTADORES E SISTEMAS DISTRIBUÍDOS (SBRC), 42. , 2024, Niterói/RJ.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 113-126.
ISSN 2177-9384.
DOI: https://doi.org/10.5753/sbrc.2024.1275.
