Multimodal LLMs as Image-to-Tabular Translators for Ultra-Short-Term Photovoltaic Forecasting

• Diego de Carvalho Neves da Fontoura UFPel https://orcid.org/0009-0006-9167-9515
• Marilton Sanchotene de Aguiar UFPel https://orcid.org/0000-0002-5247-6022

DOI: https://doi.org/10.5753/semish.2026.23777

Resumo

Ultra-short-term photovoltaic forecasting is strongly affected by cloud dynamics, yet integrating visual information typically requires complex, dedicated computer vision architectures. This study proposes an alternative approach in which multimodal LLMs (OpenAI and Gemini) serve as image-to-tabular translators. The method converts directional sky-camera images into numeric variables, which are then fused with conventional predictors, such as inverter and meteorological data. Experimental evaluations across 5-, 10-, and 15-minute horizons demonstrate that LLM-enhanced configurations consistently outperform a strong tabular baseline. The results suggest that LLMs can serve as a modular and auditable intermediate layer between raw imagery and standard forecasting models. Limitations include modest performance gains and reliance on a single pilot installation, requiring future multi-site validation.

Palavras-chave: Photovoltaic forecasting, Multimodal large language models, Sky images, Image-to-tabular translation, Time-series forecasting

Referências

Ahmed, R., Sreeram, V., Mishra, Y., and Arif, M. (2020). A review and evaluation of the state-of-the-art in pv solar power forecasting: Techniques and optimization. Renewable and Sustainable Energy Reviews, 124:109792.

Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., and Barr, I. (2022). Flamingo: a visual language model for few-shot learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Proc. of the 36th Conference on Neural Information Processing Systems (NeurIPS), volume 35, pages 23716–23736, New Orleans, USA. Curran Associates, Inc.

Antonanzas, J., Osorio, N., Escobar, R., Urraca, R., Martinez-de Pison, F., and Antonanzas-Torres, F. (2016). Review of photovoltaic power forecasting. Solar Energy, 136:78–111.

Bergmeir, C., Hyndman, R., and Koo, B. (2018). A note on the validity of cross-validation for evaluating autoregressive time series prediction. Computational Statistics & Data Analysis, 120(C):70–83.

Chow, C. W., Belongie, S., and Kleissl, J. (2015). Cloud motion and stability estimation for intra-hour solar forecasting. Solar Energy, 115:645–655.

Garniwa, P. M., Rajagukguk, R. A., Kamil, R., and Lee, H. (2023). Intraday forecast of global horizontal irradiance using optical flow method and long short-term memory model. Solar Energy, 252:234–251.

Google DeepMind (2025). Gemini 2.5 flash model. [link].

Li, J., Li, D., Savarese, S., and Hoi, S. (2023). BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proc. of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 19730–19742, Honolulu, Hawaii. PMLR.

Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2023). Visual instruction tuning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Proc. of the 37th Conference on Neural Information Processing Systems (NeurIPS), volume 36, pages 34892–34916, New Orleans, USA. Curran Associates, Inc.

OpenAI (2025). Gpt models with vision. [link].

Paletta, Q., Arbod, G., and Lasenby, J. (2023). Omnivision forecasting: Combining satellite and sky images for improved deterministic and probabilistic intra-hour solar energy predictions. Applied Energy, 336:120818.

Paletta, Q., Hu, A., Arbod, G., Blanc, P., and Lasenby, J. (2022). Spin: Simplifying polar invariance for neural networks application to vision-based irradiance forecasting. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 5182–5191, New Orleans, USA. IEEE.

Sarkis, R., Oguz, I., Psaltis, D., Paolone, M., Moser, C., and Lambertini, L. (2024). Intraday solar irradiance forecasting using public cameras. Solar Energy, 275:112600.

Sharma, E. and Elmenreich, W. (2021). A review on physical and data-driven based nowcasting methods using sky images. In Arai, K., editor, Advances in Information and Communication, pages 352–370. Springer International Publishing, Cham.

Straub, N., Herzberg, W., Dittmann, A., and Lorenz, E. (2024). Blending of a novel all sky imager model with persistence and a satellite based model for high-resolution irradiance nowcasting. Solar Energy, 269:112319.

Ukwuoma, C. C., Cai, D., Bamisile, O., Yin, H., Nneji, G. U., Monday, H. N., Oluwasanmi, A., and Huang, Q. (2024). An attention fused sequence-to-sequence convolutional neural network for accurate solar irradiance forecasting and prediction using sky images. Renewable Energy, 237:121692.

Voyant, C., Notton, G., Kalogirou, S., Nivet, M.-L., Paoli, C., Motte, F., and Fouilloy, A. (2017). Machine learning methods for solar radiation forecasting: A review. Renewable Energy, 105:569–582.

Xu, S., Zhang, R., Ma, H., Ekanayake, C., and Cui, Y. (2024). On vision transformer for ultra-short-term forecasting of photovoltaic generation using sky images. Solar Energy, 267:112203.

Zhang, L., Wilson, R., Sumner, M., and Wu, Y. (2023). Advanced multimodal fusion method for very short-term solar irradiance forecasting using sky images and meteorological data: A gate and transformer mechanism approach. Renewable Energy, 216:118952.

Zhen, Z., Liu, J., Zhang, Z., Wang, F., Chai, H., Yu, Y., Lu, X., Wang, T., and Lin, Y. (2020). Deep learning based surface irradiance mapping model for solar pv power forecasting using sky image. IEEE Transactions on Industry Applications, 56(4):3385–3396.