Análise do uso de Modelos de Linguagem de Grande Escala na Geração de Códigos para Automação Residencial - Cruz, Antonio; Rettore, Paulo H. L.; Santos, Bruno P. (Proceedings of the Brazilian Symposium on Multimedia and the Web (WebMedia); 2025: Proceedings of the 31st Brazilian Symposium on Multimedia and the WebDO - 10.5753/webmedia.2025.15971, 2025)
This paper presents a comparative experimental evaluation of Large Language Models (LLMs) for generating YAML code in home automation using the Home Assistant (HA) platform. Four models were tested: GPT-4o, GPT-4.5, Gemini 2.5 Flash, and Gemini 2.5 Pro. Tests covered three complexity levels and two prompt types. Performance was analyzed with a 2 𝑘 factorial design and validated with ANOVA. Task complexity was the main factor affecting success, while prompt specificity reduced errors. All models succeeded in basic tasks, but only Gemini 2.5 Pro maintained functionality in complex scenarios. Error analysis showed that logic errors were most frequent, followed by syntax and entity errors. A local fine-tuned model (DeepSeek-r1:14B) was also evaluated. It worked for simpler tasks but required manual adjustments. The results highlight the importance of model robustness, prompt quality, and deployment context in LLM-based code generation for home automation.