Human Activity Recognition based on Wearable Sensors using Multiscale DCNN Ensemble

• Jessica Sena UFMG
• William Robson Schwartz UFMG

DOI: https://doi.org/10.5753/sibgrapi.est.2019.8310

Resumo

Sensor-based Human Activity Recognition (HAR) provides valuable knowledge to many areas. Recently, wearable devices have gained space as a relevant source of data. However, there are two issues: large number of heterogeneous sensors available and the temporal nature of the sensor data. To handle these issues, we propose a multimodal approach that processes each sensor separately and, through an ensemble of Deep Convolution Neural Networks (DCNN), extracts information from multiple temporal scales of the sensor data. In this ensemble, we use a convolutional kernel with a different height for each DCNN. Considering that the number of rows in the sensor data reflects the data captured over time, each kernel height reflects a temporal scale from which we can extract patterns. Consequently, our approach is able to extract information from simple movement patterns such as a wrist twist when picking up a spoon, to complex movements such as the human gait. This multimodal and multi-temporal approach outperforms previous state-of-the-art works in seven important datasets using two different protocols. In addition, we demonstrate that the use of our proposed set of kernels improves sensor-based HAR in another multi-kernel approach, the widely employed inception network.

Referências

O. D. Lara and M. A. Labrador, “A survey on human activity recognition using wearable sensors.” IEEE Communications Surveys and Tutorials, 2013. https://doi.org/10.1109/SURV.2012.110112.00192

Y. Chen and Y. Xue, “A Deep Learning Approach to Human Activity Recognition Based on Single Accelerometer,” in SMC, 2015. https://doi.org/10.1109/smc.2015.263

A. Jordao, L. A. B. Torres, and W. R. Schwartz, “Novel approaches to human activity recognition based on accelerometer data,” Signal, Image and Video Processing, 2018. https://doi.org/10.1007/s11760-018-1293-x

A. Jordao, R. B. Kloss, and W. R. Schwartz, “Latent hypernet: Exploring all layers from convolutional neural networks,” in IJCNN, 2018. https://doi.org/10.1109/IJCNN.2018.8489506

W. Jiang and Z. Yin, “Human activity recognition using wearable sensors by deep convolutional neural networks,” in ACM Multimedia Conference, 2015. https://doi.org/10.1145/2733373.2806333

S. Ha, J.-M. Yun, and S. Choi, “Multi-modal convolutional neural networks for activity recognition,” in SMC, 2015. https://doi.org/10.1109/smc.2015.525

S. Ha and S. Choi, “Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors,” in IJCNN, 2016. https://doi.org/10.1109/ijcnn.2016.7727224

S. Yao, S. Hu, Y. Zhao, A. Zhang, and T. Abdelzaher, “Deepsense: A unified deep learning framework for time-series mobile sensing data processing,” 2017. https://doi.org/10.1145/3038912.3052577

K. Cho, B. van Merrienboer, Ç. Gülçehre, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” https://doi.org/10.3115/v1%2Fd14-1179

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” CoRR, 2014.

G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self-normalizing neural networks,” 2017.

D. Pedamonti, “Comparison of non-linear activation functions for deep neural networks on mnist classification task,” arXiv preprint arXiv:1804.02763, 2018.

A. Jordao, A. C. Nazare Jr, J. Sena, and W. R. Schwartz, “Human activity recognition based on wearable sensor data: A standardization of the state-of-the-art,” arXiv preprint arXiv:1806.05226, 2018.

J. R. Kwapisz, G. M. Weiss, and S. Moore, “Activity recognition using cell phone accelerometers,” SIGKDD Explorations, 2010. https://doi.org/10.1145/1964897.1964918

C. Catal, S. Tufekci, E. Pirmit, and G. Kocabag, “On the use of ensemble of classifiers for accelerometer-based activity recognition,” Applied Soft Computing, 2015. https://doi.org/10.1016/j.asoc.2015.01.025

H.-J. Kim and Y. S. Choi, “Eating activity recognition for health and wellness: A case study on asian eating style,” in ICCE, 2013. https://doi.org/10.1109/icce.2013.6486969

Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, “Efficient backprop,” in Neural networks: Tricks of the trade. Springer, 2012, pp. 9–48. https://doi.org/10.1007/978-3-642-35289-8_3

B. Bruno, F. Mastrogiovanni, and A. Sgorbissa, “Wearable Inertial Sensors: Applications, Challenges, and Public Test Benches,” in IEEE Robot. Automat. Mag., 2015. https://doi.org/10.1109/mra.2015.2448279

M. Zhang and A. A. Sawchuk, “Usc-had: A daily activity dataset for ubiquitous activity recognition using wearable sensors,” in UbiComp, 2012. https://doi.org/10.1145/2370216.2370438

C. Chen, R. Jafari, and N. Kehtarnavaz, “UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor,” in ICIP, 2015. https://doi.org/10.1109/icip.2015.7350781

J. W. Lockhart, G. M. Weiss, J. C. Xue, S. T. Gallagher, A. B. Grosner, and T. T. Pulickal, “Design considerations for the wisdm smart phone-based sensor mining architecture,” in SensorKDD, 2011. https://doi.org/10.1145/2003653.2003656

A. Reiss and D. Stricker, “Introducing a new benchmarked dataset for activity monitoring,” in ISWC, 2012. https://doi.org/10.1109/iswc.2012.13

O. Baños, R. Garcı́a, J. A. Holgado-Terriza, M. Damas, H. Pomares, I. R. Ruiz, A. Saez, and C. Villalonga, “mhealthdroid: A novel framework for agile development of mobile health applications,” in IWAAL, 2014. https://doi.org/10.1007/978-3-319-13105-4_14

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR09, 2009. https://doi.org/10.1109/CVPRW.2009.5206848

J. Sena, J. B. Santos, and W. R. Schwartz, “Multiscale dcnn ensemble applied to human activity recognition based on wearable sensors,” in Signal Processing Conference (EUSIPCO), 2018 Proceedings of the 26th European. IEEE, 2018. https://doi.org/10.23919/eusipco.2018.8553354