Vision-Language Models for Automated Property Feature Extraction in Tax Assessment: A Case Study - Ribeiro, Gustavo R.; Saraiva, Pedro A. M.; Rosa, Luis H. A.; Marques, Enzo L.; Pereira, Gustavo L. B.; Campos, Pedro M. L.; Pascoal, Luiz M. L.; Oliveira, Sávio S. T. de (Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC); 2025: Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional, 2025)
The calculation of property taxes, such as the Urban Building and Land Tax (IPTU) in Brazil, is a critical function for municipal revenue. This process traditionally relies on manual, on-site inspections to assess property characteristics, a task that is costly, time-consuming, and prone to subjectivity. This paper explores the potential of automating this process through the application of state-of-the-art Vision-Language Models (VLMs). We present a novel benchmark to evaluate the capabilities of twelve different VLMs in identifying and classifying specific building features as defined by the municipal legislation of Goiânia, Brazil. Using a dataset of images from public real estate listings and a zero-shot prompting strategy, we tasked the models with extracting 11 distinct construction categories, such as flooring, structure, and finishes. Our results indicate that proprietary models, particularly Google’s Gemini 1.5 Pro and Gemini 1.5 Flash, achieve the highest performance, with macro F1-scores of 0.77 and 0.76, respectively. We provide a detailed analysis of model performance across different categories, revealing that while some features like ’Structure’ and ’Electrical Installation’ are identified with high accuracy, others like ’Sanitary Installation’ and ’External Finishes’ remain challenging due to their visual subtlety or absence in typical photographs. Our findings demonstrate the significant potential of VLMs to streamline public administration tasks, while also highlighting current limitations and avenues for future research.