Multimodal Fusion Strategies for Multivariate Time Series Classification - Bastos, Samuel Thomaz; Silva, Rafael da Costa; Silva, Diego Furtado (Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC); 2025: Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional, 2025)
Time series classification (TSC) is relevant in various domains, including finance, healthcare, and human activity recognition. While deep learning methods have achieved state-of-the-art performance in TSC, most advancements focus on univariate or straightforward adaptations for multivariate time series, neglecting different inter-variable dependencies. This problem increases in scenarios where the data comes from multiple sensors. In this case, the classifier does not consider the particularities of each source. For instance, consider a motion recognition dataset containing data from accelerometers and gyroscopes. A simple architecture for multivariate TSC applies the same transformations, e.g., kernel convolutions, to both sensors, even if they naturally present different characteristics. This paper argues that treating the variables of distinct sources as separate modalities, akin to sensor data, and employing specialized models for each source can enhance the classification performance. To evaluate this claim, we propose treating sensor inputs as multimodal data, applying fusion techniques to integrate multi-source multivariate time series classifiers. Experimental analysis shows that our approach outperforms traditional singlemodel architectures, particularly in complex tasks. In these worst scenarios for the single-modality models, our proposal achieves accuracy up to twelve percentage points higher, achieving the best results in the literature. It highlights the importance of considering multi-source dependencies in TSC, highlighting the potential of fusion-based strategies to advance the multivariate time series classification field.