A Reliable and Low-Latency Graph-Routing Approach for IWSN using Q-Routing
Resumo
For Industrial Wireless Sensor Networks, the Network Manager is responsible for the overall configuration, route definition, allocation of communication resources, and optimization of the network. Graph routing is used to increase the reliability of the network through path redundancy. Reinforcement Learning models have been used to optimize latency, energy consumption, and data delivery. Q-Routing is a learning model where nodes in the network learn which of its neighboring nodes provide the best routes for a destination node. Despite presenting good results, this model is not applicable to centralized networks since it does not provide path redundancy and nodes are not allowed to choose routes. We present the Q-Learning Reliable Routing with Multiple Agents approach, that builds routing graphs in a centralized manner using Q-Routing. Each node is represented by a learning agent. Periodically, each agent acts by choosing neighbors used to forward data to a destination. An updated graph is then built and configured over the network. Rewards are given to each agent when its average data latency decreases. Simulations were conducted on a WirelessHART simulator. Results show, in most cases, a reduction of the average network latency while the communication reliability is at least as good as the state-of-the-art graph-routing algorithms.
Palavras-chave:
Industrial Wireless Sensor Networks, Routing, Reinforcement Learning, Q-Routing, WirelessHART
Referências
M. Sha D. Gunatilaka C. Wu and C. Lu "Empirical study and enhancements of industrial wireless sensor-actuator network protocols" IEEE Internet of Things Journal vol. 4 no. 3 pp. 696-704 June 2017.
GVR 2018 [online] Available: https://www.grandviewresearch.com/press-release/global-industrial-wireless-sensor-networks-iwsn-market.
J. M. Winter I. Muller G. Soatti S. Savazzi M. Nicoli L. B. Becker et al. "Wireless coexistence and spectrum sensing in industrial nternet of things: An experimental study" International Journal of Distributed Sensor Networks 2015.
G. Kunzel L. S. Indrusiak and C. E. Pereira "Latency and lifetime enhancements in industrial wireless sensor networks: A q-learning approach for graph routing" IEEE Transactions on Industrial Informatics vol. 16 no. 8 pp. 5617-5625 2020.
W. Shen T. Zhang F. Barac and M. Gidlund "Prioritymac: A priority-enhanced mac protocol for critical traffic in industrial wireless sensor and actuator networks" IEEE Transactions on Industrial Informatics vol. 10 no. 1 pp. 824-835 Feb 2014.
M. Nobre I. Silva and L. A. Guedes "Routing and scheduling algorithms for wirelesshart networks: A survey" Sensors vol. 15 no. 5 pp. 9703-9740 2015.
S. Han X. Zhu A. K. Mok D. Chen and M. Nixon "Reliable and real-time communication in industrial wireless mesh networks" 2011 17th IEEE Real- Time and Embedded Technology and Applications Symposium pp. 3-12 April 2011.
Z. Jindong L. Zhenjun and Z. Yaopei "Elhfr: A graph routing in industrial wireless mesh network" 2009 International Conference on Information and Automation pp. 106-110 June 2009.
G. Kunzel G. P. Cainelli and C. E. Pereira "A weighted broadcast routing algorithm for wirelesshart networks" 2017 VII Brazilian Symposium on Computing Systems Engineering (SBESC) pp. 187-192 Nov. 2017.
S. Zhang A. Yan and T. Ma "Energy-balanced routing for maximizing network lifetime in wirelesshart" International Journal of Distributed Sensor Networks vol. 9 no. 10 2013.
Q. Zhang F. Li L. Ju Z. Jia and Z. Zhang Reliable and Energy Efficient Routing Algorithm for WirelessHART Cham: Springer International Publishing pp. 192-203 2014.
C. Wu D. Gunatilaka A. Saifullah M. Sha P. B. Tiwari C. Lu et al. "Maximizing network lifetime of wirelesshart networks under graph routing" 2016 IEEE First International Conference on Internet-of-Things Design and Implementation (IoTDI) pp. 176-186 April 2016.
M. Sepulcre J. Gozalvez and B. Coll-Perales "Multipath qos-driven routing protocol for industrial wireless networks" Journal of Network and Computer Applications vol. 74 no. Supplement C pp. 121-132 2016.
X. Han X. Ma and D. Chen "Energy-balancing routing algorithm for wirelesshart" 2019 15th IEEE International Workshop on Factory Communication Systems (WFCS) pp. 1-7 May 2019.
R. S. Sutton and A. G. Barto Reinforcement Learning: An Introduction The MIT Press 2018.
C. Savaglio P. Pasquale G. Aloi A. Liotta and G. Fortino "Lightweight reinforcement learning for energy efficient communications in wireless sensor networks" IEEE Access vol. PP pp. 1-1 03 2019.
Z. Mammeri "Reinforcement learning based routing in networks: Review and classification of approaches" IEEE Access vol. 7 pp. 55916-55950 2019.
H. A. Al-Rawi M. A. Ng and K.-L. A. Yau "Application of reinforcement learning to routing in distributed wireless networks: A review" Artif. Intell. Rev. vol. 43 no. 3 pp. 381-416 Mar. 2015.
M. A. Habib M. Y. Arafat and S. Moh "Routing protocols based on reinforcement learning for wireless sensor networks: A comparative study" Journal of Advanced Research in Dynamical and Control Systems pp. 427-435 2019.
P. Zand E. Mathews P. Havinga S. Stojanovski E. Sisinni and P. Ferrari "Implementation of wirelesshart in the ns-2 simulator and validation of its correctness" Sensors vol. 14 no. 5 pp. 8633-8668 2014.
W. Guo C. Yan and T. Lu "Optimizing the lifetime of wireless sensor networks via reinforcement-learning-based routing" International Journal of Distributed Sensor Networks vol. 15 no. 2 2019.
Z. Jin Y. Ma Y. Su S. Li and X. Fu "A q-learning-based delay-aware routing algorithm to extend the lifetime of underwater sensor networks" Sensors vol. 17 no. 7 2017.
S. Koenig and R. G. Simmons Complexity analysis of real-time reinforcement learning applied to finding shortest paths in deterministic domains 1992.
"HCF HCF_SPEC-290: WirelessHART Device Specification" Hart Communication Foundation 2008.
A. Bildea O. Alphand F. Rousseau and A. Duda "Link quality metrics in large scale indoor wireless sensor networks" 2013 IEEE 24th Annual International Symposium on Personal Indoor and Mobile Radio Communications (PIMRC) pp. 1888-1892 Sep. 2013.
J. Akerberg M. Gidlund and M. Bjorkman "Future research challenges in wireless sensor and actuator networks targeting industrial automation" 2011 9th IEEE International Conference on Industrial Informatics pp. 410-415 July 2011.
GVR 2018 [online] Available: https://www.grandviewresearch.com/press-release/global-industrial-wireless-sensor-networks-iwsn-market.
J. M. Winter I. Muller G. Soatti S. Savazzi M. Nicoli L. B. Becker et al. "Wireless coexistence and spectrum sensing in industrial nternet of things: An experimental study" International Journal of Distributed Sensor Networks 2015.
G. Kunzel L. S. Indrusiak and C. E. Pereira "Latency and lifetime enhancements in industrial wireless sensor networks: A q-learning approach for graph routing" IEEE Transactions on Industrial Informatics vol. 16 no. 8 pp. 5617-5625 2020.
W. Shen T. Zhang F. Barac and M. Gidlund "Prioritymac: A priority-enhanced mac protocol for critical traffic in industrial wireless sensor and actuator networks" IEEE Transactions on Industrial Informatics vol. 10 no. 1 pp. 824-835 Feb 2014.
M. Nobre I. Silva and L. A. Guedes "Routing and scheduling algorithms for wirelesshart networks: A survey" Sensors vol. 15 no. 5 pp. 9703-9740 2015.
S. Han X. Zhu A. K. Mok D. Chen and M. Nixon "Reliable and real-time communication in industrial wireless mesh networks" 2011 17th IEEE Real- Time and Embedded Technology and Applications Symposium pp. 3-12 April 2011.
Z. Jindong L. Zhenjun and Z. Yaopei "Elhfr: A graph routing in industrial wireless mesh network" 2009 International Conference on Information and Automation pp. 106-110 June 2009.
G. Kunzel G. P. Cainelli and C. E. Pereira "A weighted broadcast routing algorithm for wirelesshart networks" 2017 VII Brazilian Symposium on Computing Systems Engineering (SBESC) pp. 187-192 Nov. 2017.
S. Zhang A. Yan and T. Ma "Energy-balanced routing for maximizing network lifetime in wirelesshart" International Journal of Distributed Sensor Networks vol. 9 no. 10 2013.
Q. Zhang F. Li L. Ju Z. Jia and Z. Zhang Reliable and Energy Efficient Routing Algorithm for WirelessHART Cham: Springer International Publishing pp. 192-203 2014.
C. Wu D. Gunatilaka A. Saifullah M. Sha P. B. Tiwari C. Lu et al. "Maximizing network lifetime of wirelesshart networks under graph routing" 2016 IEEE First International Conference on Internet-of-Things Design and Implementation (IoTDI) pp. 176-186 April 2016.
M. Sepulcre J. Gozalvez and B. Coll-Perales "Multipath qos-driven routing protocol for industrial wireless networks" Journal of Network and Computer Applications vol. 74 no. Supplement C pp. 121-132 2016.
X. Han X. Ma and D. Chen "Energy-balancing routing algorithm for wirelesshart" 2019 15th IEEE International Workshop on Factory Communication Systems (WFCS) pp. 1-7 May 2019.
R. S. Sutton and A. G. Barto Reinforcement Learning: An Introduction The MIT Press 2018.
C. Savaglio P. Pasquale G. Aloi A. Liotta and G. Fortino "Lightweight reinforcement learning for energy efficient communications in wireless sensor networks" IEEE Access vol. PP pp. 1-1 03 2019.
Z. Mammeri "Reinforcement learning based routing in networks: Review and classification of approaches" IEEE Access vol. 7 pp. 55916-55950 2019.
H. A. Al-Rawi M. A. Ng and K.-L. A. Yau "Application of reinforcement learning to routing in distributed wireless networks: A review" Artif. Intell. Rev. vol. 43 no. 3 pp. 381-416 Mar. 2015.
M. A. Habib M. Y. Arafat and S. Moh "Routing protocols based on reinforcement learning for wireless sensor networks: A comparative study" Journal of Advanced Research in Dynamical and Control Systems pp. 427-435 2019.
P. Zand E. Mathews P. Havinga S. Stojanovski E. Sisinni and P. Ferrari "Implementation of wirelesshart in the ns-2 simulator and validation of its correctness" Sensors vol. 14 no. 5 pp. 8633-8668 2014.
W. Guo C. Yan and T. Lu "Optimizing the lifetime of wireless sensor networks via reinforcement-learning-based routing" International Journal of Distributed Sensor Networks vol. 15 no. 2 2019.
Z. Jin Y. Ma Y. Su S. Li and X. Fu "A q-learning-based delay-aware routing algorithm to extend the lifetime of underwater sensor networks" Sensors vol. 17 no. 7 2017.
S. Koenig and R. G. Simmons Complexity analysis of real-time reinforcement learning applied to finding shortest paths in deterministic domains 1992.
"HCF HCF_SPEC-290: WirelessHART Device Specification" Hart Communication Foundation 2008.
A. Bildea O. Alphand F. Rousseau and A. Duda "Link quality metrics in large scale indoor wireless sensor networks" 2013 IEEE 24th Annual International Symposium on Personal Indoor and Mobile Radio Communications (PIMRC) pp. 1888-1892 Sep. 2013.
J. Akerberg M. Gidlund and M. Bjorkman "Future research challenges in wireless sensor and actuator networks targeting industrial automation" 2011 9th IEEE International Conference on Industrial Informatics pp. 410-415 July 2011.
Publicado
23/11/2020
Como Citar
KÜNZEL, Gustavo; CAINELLI, Gustavo; MÜLLER, Ivan; PEREIRA, Carlos Eduardo; INDRUSIAK, Leandro Soares.
A Reliable and Low-Latency Graph-Routing Approach for IWSN using Q-Routing. In: SIMPÓSIO BRASILEIRO DE ENGENHARIA DE SISTEMAS COMPUTACIONAIS (SBESC), 10. , 2020, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 1-8.
ISSN 2237-5430.
