Named Entity Extraction in Industrial Diagrams with Computer Vision and MLLMs - Weibull, Jon Karl; Santos, Filip Duarte dos; Souza, Jefferson Alves de; Lima, Denyson Tomaz de; Lemos, Melissa; Casanova, Marco Antonio (Anais Estendidos da Conference on Graphics, Patterns and Images (SIBGRAPI); 2025: Anais Estendidos da XXXVIII Conference on Graphics, Patterns and ImagesDO - 10.5753/sibgrapi.est.2025.38319, 2025)
This paper presents a comprehensive evaluation of methods for extracting named entities (tags) from Piping and Instrumentation Diagrams (P&IDs), critical documents used throughout the lifecycle of industrial facilities. Efficient indexing and retrieval of information from such technical documentation play a key role in improving industrial project management. The paper proposes integrating deep neural networks for object detection with state-of-the-art Multimodal Large Language Models (MLLMs), specifically GPT-4o and LLaMA-3.2-Vision-90B. The methodology includes a preprocessing stage to isolate and downscale symbols before tag extraction. Experimental results on a challenging dataset of 30 real-world symbols show that MLLM-based approaches significantly outperform traditional OCR techniques (82%), achieving near-perfect accuracy in most cases. The approach demonstrates the robustness of MLLMs under noisy and degraded conditions, offering a practical solution for augmenting engineering document databases with reliable named entities. These findings highlight the potential of MLLMs to enable robust, high-precision tag text extraction, thereby streamlining the indexing and management of complex engineering documentation. All 30 test symbols used in this study are provided as supplementary material to ensure reproducibility.