Towards novel smart wearable sensors to classify subject-specific human walking activities
Resumo
In this century, smart devices are increasingly present in our lives, such as at work, sports, or household chores. In this context, we have wearable devices that can help people with health monitoring or physical performance in sports activities. With the integration of artificial intelligence (AI), these wearable devices can identify injuries in athletes or care for the elderly in rehabilitation from human activity recognition (HAR). AI techniques are commonly applied for pattern recognition, such as image classification or HAR. In this context, we seek to develop a smart wearable device to recognize walking activities. In order to improve the identification of these tasks through AI algorithms, we propose the fusion of data between four sensors called SPUs. Each SPU has NodeMCU ESP-32 and BNO080 IMU hardware in its architecture. The data from these hardware provides information in high precision. A zero W raspberry pi collected this information. After extracting and manipulating this data, we trained a deep learning model. The model accuracy was higher than 92% reaching an overall accuracy of 97%. Therefore, the smart wearable device showed a new tool for recognizing walking activity, which could be applied in the future to recognize more complex tasks.
Palavras-chave:
HAR, LSTM, Wearable, Sensors, Walk, AI
Referências
J. Heikenfeld, A. Jajack, J. Rogers, P. Gutruf, L. Tian, T. Pan, R. Li, M. Khine, J. Kim, and J. Wang, “Wearable sensors: modalities, challenges, and prospects,” Lab on a Chip, vol. 18, no. 2, pp. 217–248, 2018.
O. Yurur and C. H. Liu, Generic and energy-efficient context-aware mobile sensing. CRC Press, 2015.
S. Patel, H. Park, P. Bonato, L. Chan, and M. Rodgers, “A review of wearable sensors and systems with application in rehabilitation,” Journal of neuroengineering and rehabilitation, vol. 9, no. 1, pp. 1–17, 2012.
H. F. Nweke, Y. W. Teh, M. A. Al-garadi, and U. R. Alo, “Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges,” Expert Systems with Applications, vol. 105, pp. 233–261, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417418302136
M. Masoud, Y. Jaradat, A. Manasrah, and I. Jannoud, “Sensors of smart devices in the internet of everything (ioe) era: big opportunities and massive doubts,” Journal of Sensors, vol. 2019, 2019.
M. Cornacchia, K. Ozcan, Y. Zheng, and S. Velipasalar, “A survey on activity detection and classification using wearable sensors,” IEEE Sensors Journal, vol. 17, no. 2, pp. 386–403, 2017.
I. M. Pires, N. M. Garcia, N. Pombo, and F. Florez-Revuelta, “From ´ data acquisition to data fusion: a comprehensive review and a roadmap for the identification of activities of daily living using mobile devices,” Sensors, vol. 16, no. 2, p. 184, 2016.
R. Gravina, P. Alinia, H. Ghasemzadeh, and G. Fortino, “Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges,” Information Fusion, vol. 35, pp. 68–80, 2017.
M. Shoaib, S. Bosch, O. D. Incel, H. Scholten, and P. J. Havinga, “Fusion of smartphone motion sensors for physical activity recognition,” Sensors, vol. 14, no. 6, pp. 10 146–10 176, 2014.
Y. Silina and H. Haddadi, “New directions in jewelry: A close look at emerging trends & developments in jewelry-like wearable devices,” p. 49–56, 2015. [Online]. Available: https://doi.org/10.1145/2802083.2808410
V. J. P. Amorim, R. A. R. Oliveira, and M. J. da Silva, “Recent trends in wearable computing research: A systematic review,” 2020.
V. J. P. Amorim, M. C. Silva, and R. A. R. Oliveira, “Software and hardware requirements and trade-offs in operating systems for wearables: A tool to improve devices’ performance,” April 2019.
B. P. Kirby and B. Mosley, “The architecture of wearable technology,” 2015.
R. Younes, “Improving the accuracy of wearable activity classifiers,” p. 509–514, 2015. [Online]. Available: https://doi.org/10.1145/2800835.2801656
M. E. Berglund, J. Duvall, and L. E. Dunne, “A survey of the historical scope and current trends of wearable technology applications,” p. 40–43, 2016. [Online]. Available: https://doi.org/10.1145/2971763.2971796
H. Raad, Fundamentals of IoT and Wearable Technology Design. Wiley-IEEE Press, 2021.
J. A. E. Isasa, P. G. Larsen, and F. O. Hansen, “Energy-aware model driven development of a wearable healthcare device,” pp. 44–63, 2017.
S. Hiremath, G. Yang, and K. Mankodiya, “Wearable internet of things: Concept, architectural components and promises for person-centered healthcare,” pp. 304–307, 2014.
J. Gubbi, R. Buyya, S. Marusic, and M. Palaniswami, “Internet of things (iot): A vision, architectural elements, and future directions,” Future Generation Computer Systems, vol. 29, no. 7, pp. 1645–1660, 2013. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167739X13000241
S. Mekruksavanich and A. Jitpattanakul, “Biometric user identification based on human activity recognition using wearable sensors: An experiment using deep learning models,” Electronics, vol. 10, no. 3, 2021. [Online]. Available: https://www.mdpi.com/2079-9292/10/3/308
R. Romijnders, E. Warmerdam, C. Hansen, G. Schmidt, and W. Maetzler, “A deep learning approach for gait event detection from a single shank-worn imu: Validation in healthy and neurological cohorts,” Sensors, vol. 22, no. 10, 2022. [Online]. Available: https://www.mdpi.com/1424-8220/22/10/3859
Y. Wang, S. Cang, and H. Yu, “A survey on wearable sensor modality centred human activity recognition in health care,” Expert Systems with Applications, vol. 137, pp. 167–190, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417419302878
Y. Celik, S. Stuart, W. L. Woo, and A. Godfrey, “Wearable inertial gait algorithms: Impact of wear location and environment in healthy and parkinson’s populations,” Sensors, vol. 21, no. 19, 2021. [Online]. Available: https://www.mdpi.com/1424-8220/21/19/6476
J. Lu, X. Zheng, M. Sheng, J. Jin, and S. Yu, “Efficient human activity recognition using a single wearable sensor,” IEEE Internet of Things Journal, vol. 7, no. 11, pp. 11 137–11 146, 2020.
J. Chen and X. Ran, “Deep learning with edge computing: A review,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1655–1674, 2019.
S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar, and A. Zomaya, “Edge intelligence: The confluence of edge computing and artificial intelligence,” 09 2019.
L. Liu, H. Li, and M. Gruteser, “Edge assisted real-time object detection for mobile augmented reality,” The 25th Annual International Conference on Mobile Computing and Networking, 2019.
G. Santos, T. Tavares, and A. Rocha, “Reliability and generalization of gait biometrics using 3d inertial sensor data and 3d optical system trajectories,” Scientific Reports, vol. 12, no. 1, pp. 1–15, 2022.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
A. Graves, “Long short-term memory,” Supervised sequence labelling with recurrent neural networks, pp. 37–45, 2012.
“Dataset for human activity recognition har,” [link].
N. Ravi, N. Dandekar, P. Mysore, and M. L. Littman, “Activity recognition from accelerometer data,” in Aaai, vol. 5, no. 2005. Pittsburgh, PA, 2005, pp. 1541–1546.
M. Hossin and M. N. Sulaiman, “A review on evaluation metrics for data classification evaluations,” International journal of data mining & knowledge management process, vol. 5, no. 2, p. 1, 2015.
O. Yurur and C. H. Liu, Generic and energy-efficient context-aware mobile sensing. CRC Press, 2015.
S. Patel, H. Park, P. Bonato, L. Chan, and M. Rodgers, “A review of wearable sensors and systems with application in rehabilitation,” Journal of neuroengineering and rehabilitation, vol. 9, no. 1, pp. 1–17, 2012.
H. F. Nweke, Y. W. Teh, M. A. Al-garadi, and U. R. Alo, “Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges,” Expert Systems with Applications, vol. 105, pp. 233–261, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417418302136
M. Masoud, Y. Jaradat, A. Manasrah, and I. Jannoud, “Sensors of smart devices in the internet of everything (ioe) era: big opportunities and massive doubts,” Journal of Sensors, vol. 2019, 2019.
M. Cornacchia, K. Ozcan, Y. Zheng, and S. Velipasalar, “A survey on activity detection and classification using wearable sensors,” IEEE Sensors Journal, vol. 17, no. 2, pp. 386–403, 2017.
I. M. Pires, N. M. Garcia, N. Pombo, and F. Florez-Revuelta, “From ´ data acquisition to data fusion: a comprehensive review and a roadmap for the identification of activities of daily living using mobile devices,” Sensors, vol. 16, no. 2, p. 184, 2016.
R. Gravina, P. Alinia, H. Ghasemzadeh, and G. Fortino, “Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges,” Information Fusion, vol. 35, pp. 68–80, 2017.
M. Shoaib, S. Bosch, O. D. Incel, H. Scholten, and P. J. Havinga, “Fusion of smartphone motion sensors for physical activity recognition,” Sensors, vol. 14, no. 6, pp. 10 146–10 176, 2014.
Y. Silina and H. Haddadi, “New directions in jewelry: A close look at emerging trends & developments in jewelry-like wearable devices,” p. 49–56, 2015. [Online]. Available: https://doi.org/10.1145/2802083.2808410
V. J. P. Amorim, R. A. R. Oliveira, and M. J. da Silva, “Recent trends in wearable computing research: A systematic review,” 2020.
V. J. P. Amorim, M. C. Silva, and R. A. R. Oliveira, “Software and hardware requirements and trade-offs in operating systems for wearables: A tool to improve devices’ performance,” April 2019.
B. P. Kirby and B. Mosley, “The architecture of wearable technology,” 2015.
R. Younes, “Improving the accuracy of wearable activity classifiers,” p. 509–514, 2015. [Online]. Available: https://doi.org/10.1145/2800835.2801656
M. E. Berglund, J. Duvall, and L. E. Dunne, “A survey of the historical scope and current trends of wearable technology applications,” p. 40–43, 2016. [Online]. Available: https://doi.org/10.1145/2971763.2971796
H. Raad, Fundamentals of IoT and Wearable Technology Design. Wiley-IEEE Press, 2021.
J. A. E. Isasa, P. G. Larsen, and F. O. Hansen, “Energy-aware model driven development of a wearable healthcare device,” pp. 44–63, 2017.
S. Hiremath, G. Yang, and K. Mankodiya, “Wearable internet of things: Concept, architectural components and promises for person-centered healthcare,” pp. 304–307, 2014.
J. Gubbi, R. Buyya, S. Marusic, and M. Palaniswami, “Internet of things (iot): A vision, architectural elements, and future directions,” Future Generation Computer Systems, vol. 29, no. 7, pp. 1645–1660, 2013. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167739X13000241
S. Mekruksavanich and A. Jitpattanakul, “Biometric user identification based on human activity recognition using wearable sensors: An experiment using deep learning models,” Electronics, vol. 10, no. 3, 2021. [Online]. Available: https://www.mdpi.com/2079-9292/10/3/308
R. Romijnders, E. Warmerdam, C. Hansen, G. Schmidt, and W. Maetzler, “A deep learning approach for gait event detection from a single shank-worn imu: Validation in healthy and neurological cohorts,” Sensors, vol. 22, no. 10, 2022. [Online]. Available: https://www.mdpi.com/1424-8220/22/10/3859
Y. Wang, S. Cang, and H. Yu, “A survey on wearable sensor modality centred human activity recognition in health care,” Expert Systems with Applications, vol. 137, pp. 167–190, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417419302878
Y. Celik, S. Stuart, W. L. Woo, and A. Godfrey, “Wearable inertial gait algorithms: Impact of wear location and environment in healthy and parkinson’s populations,” Sensors, vol. 21, no. 19, 2021. [Online]. Available: https://www.mdpi.com/1424-8220/21/19/6476
J. Lu, X. Zheng, M. Sheng, J. Jin, and S. Yu, “Efficient human activity recognition using a single wearable sensor,” IEEE Internet of Things Journal, vol. 7, no. 11, pp. 11 137–11 146, 2020.
J. Chen and X. Ran, “Deep learning with edge computing: A review,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1655–1674, 2019.
S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar, and A. Zomaya, “Edge intelligence: The confluence of edge computing and artificial intelligence,” 09 2019.
L. Liu, H. Li, and M. Gruteser, “Edge assisted real-time object detection for mobile augmented reality,” The 25th Annual International Conference on Mobile Computing and Networking, 2019.
G. Santos, T. Tavares, and A. Rocha, “Reliability and generalization of gait biometrics using 3d inertial sensor data and 3d optical system trajectories,” Scientific Reports, vol. 12, no. 1, pp. 1–15, 2022.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
A. Graves, “Long short-term memory,” Supervised sequence labelling with recurrent neural networks, pp. 37–45, 2012.
“Dataset for human activity recognition har,” [link].
N. Ravi, N. Dandekar, P. Mysore, and M. L. Littman, “Activity recognition from accelerometer data,” in Aaai, vol. 5, no. 2005. Pittsburgh, PA, 2005, pp. 1541–1546.
M. Hossin and M. N. Sulaiman, “A review on evaluation metrics for data classification evaluations,” International journal of data mining & knowledge management process, vol. 5, no. 2, p. 1, 2015.
Publicado
21/11/2022
Como Citar
DA SILVA, Jonathan Cristovão Ferreira; DE AMORIM, Vicente José Peixoto; LAZARONI, Pedro Sebastião de Oliveira; OLIVEIRA, Ricardo Augusto Rabelo; SILVA, Mateus Coelho.
Towards novel smart wearable sensors to classify subject-specific human walking activities. In: TRABALHOS EM ANDAMENTO - SIMPÓSIO BRASILEIRO DE ENGENHARIA DE SISTEMAS COMPUTACIONAIS (SBESC), 12. , 2022, Fortaleza/CE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 68-73.
ISSN 2763-9002.
DOI: https://doi.org/10.5753/sbesc_estendido.2022.228153.
