Enhanced Forecasting Model Using Transformers, Extended Long-Short-Term Memory, and Randomized Fuzzy Cognitive Maps

• Omid Orang UFMG
• Petrônio C. L. Silva IFNMG / UFMG
• Frederico G. Guimarães UFMG

Resumo

This paper presents TxL-RHFCM, a cutting-edge hybrid deep learning model designed for time series forecasting. The architecture seamlessly combines Transformers, extended long-short-term memory (xLSTM), and randomized high-order fuzzy cognitive maps (R-HFCM), with the latter serving as a reservoir computing mechanism. The model architecture is structured into four key components. Initially, the Transformer block processes the raw time series data, employing attention mechanisms to uncover intricate relationships between sequence elements and produce refined representations. These transformed sequences are then passed to the xLSTM unit to compute hidden states from the final tokens. Subsequently, the R-HFCM module leverages these hidden states to derive concept activation states. Finally, a feed-forward network extracts meaningful features from the R-HFCM output to generate the forecast. The proposed TxL-RHFCM model is rigorously evaluated on six benchmark case studies to predict electrical load consumption. Experimental results demonstrate its superior predictive accuracy compared to various state-of-the-art deep learning, machine learning, and fuzzy time series approaches.