Multimodal Fusion Strategies for Multivariate Time Series Classification

• Samuel Thomaz Bastos USP
• Rafael da Costa Silva USP
• Diego Furtado Silva USP

DOI: https://doi.org/10.5753/eniac.2025.14459

Resumo

Time series classification (TSC) is relevant in various domains, including finance, healthcare, and human activity recognition. While deep learning methods have achieved state-of-the-art performance in TSC, most advancements focus on univariate or straightforward adaptations for multivariate time series, neglecting different inter-variable dependencies. This problem increases in scenarios where the data comes from multiple sensors. In this case, the classifier does not consider the particularities of each source. For instance, consider a motion recognition dataset containing data from accelerometers and gyroscopes. A simple architecture for multivariate TSC applies the same transformations, e.g., kernel convolutions, to both sensors, even if they naturally present different characteristics. This paper argues that treating the variables of distinct sources as separate modalities, akin to sensor data, and employing specialized models for each source can enhance the classification performance. To evaluate this claim, we propose treating sensor inputs as multimodal data, applying fusion techniques to integrate multi-source multivariate time series classifiers. Experimental analysis shows that our approach outperforms traditional singlemodel architectures, particularly in complex tasks. In these worst scenarios for the single-modality models, our proposal achieves accuracy up to twelve percentage points higher, achieving the best results in the literature. It highlights the importance of considering multi-source dependencies in TSC, highlighting the potential of fusion-based strategies to advance the multivariate time series classification field.

Referências

Ay, E., Devanne, M., Weber, J., and Forestier, G. (2022). A study of knowledge distillation in fully convolutional network for time series classification. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.

Bagnall, A., Dau, H. A., Lines, J., Flynn, M., Large, J., Bostrom, A., Southam, P., and Keogh, E. (2018). The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075.

Fawaz, H. I., Forestier, G., Weber, J., Idoumghar, L., and Muller, P.-A. (2019). Adversarial attacks on deep neural networks for time series classification. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.

Foumani, S. N. M., Tan, C. W., and Salehi, M. (2021). Disjoint-cnn for multivariate time series classification. In 2021 International Conference on Data Mining Workshops (ICDMW), pages 760–769. IEEE.

Ismail-Fawaz, A., Devanne, M., Berretti, S., Weber, J., and Forestier, G. (2025). Look into the lite in deep learning for time series classification. International Journal of Data Science and Analytics, pages 1–21.

Ismail-Fawaz, A., Devanne, M., Weber, J., and Forestier, G. (2022). Deep learning for time series classification using new hand-crafted convolution filters. In 2022 IEEE International Conference on Big Data (Big Data), pages 972–981. IEEE.

Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., and Muller, P.-A. (2019). Deep neural network ensembles for time series classification. arXiv e-prints, pages arXiv– 1903.

Ismail Fawaz, H., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D. F., Weber, J., Webb, G. I., Idoumghar, L., Muller, P.-A., and Petitjean, F. (2020). Inceptiontime: Find ing alexnet for time series classification. Data Mining and Knowledge Discovery, 34(6):1936–1962.

Lahat, D., Adali, T., and Jutten, C. (2015). Multimodal data fusion: an overview of methods, challenges, and prospects. Proceedings of the IEEE, 103(9):1449–1477.

Medeiros Júnior, J. G. B., Mitri, A. G., and Silva, D. F. (2024). Semi-periodic activation for time series classification. In Proceedings of the 34th Brazilian Conference on Intelligent Systems.

Middlehurst, M., Schäfer, P., and Bagnall, A. (2024). Bake off redux: a review and experimental evaluation of recent time series classification algorithms. Data Mining and Knowledge Discovery, pages 1–74.

Mohammadi Foumani, N., Miller, L., Tan, C. W., Webb, G. I., Forestier, G., and Salehi, M. (2024). Deep learning for time series classification and extrinsic regression: A current survey. ACM Computing Surveys, 56(9):1–45.

Pham, A.-D., Kuestenmacher, A., and Ploeger, P. G. (2023). Tsem: Temporally-weighted spatiotemporal explainable neural network for multivariate time series. In Future of Information and Communication Conference, pages 183–204. Springer.

Rakhshani, H., Fawaz, H. I., Idoumghar, L., Forestier, G., Lepagnot, J., Weber, J., Brévilliers, M., and Muller, P.-A. (2020). Neural architecture search for time series classification. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.

Ruiz, A. P., Flynn, M., Large, J., Middlehurst, M., and Bagnall, A. (2021). The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, 35(2):401–449.

Schäfer, P. and Leser, U. (2017). Multivariate time series classification with weasel+ muse. arXiv preprint arXiv:1711.11343.

Shi, P., Ye, W., and Qin, Z. (2021). Self-supervised pre-training for time series classification. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.

Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2017). Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31.

Tripathi, A. M. and Baruah, R. D. (2020). Multivariate time series classification with an attention-based multivariate convolutional neural network. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.

Wang, Z., Yan, W., and Oates, T. (2017). Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN), pages 1578–1585. IEEE.

Yeh, C.-C. M., Zhu, Y., Ulanova, L., Begum, N., Ding, Y., Dau, H. A., Zimmerman, Z., Silva, D. F., Mueen, A., and Keogh, E. (2018). Time series joins, motifs, discords and shapelets: a unifying view that exploits the matrix profile. Data Mining and Knowledge Discovery, 32(1):83–123.