Multimodal Pictogram Embeddings for Augmentative and Alternative Communication: A Comparative User Study
Resumo
Augmentative and Alternative Communication (AAC) systems rely on retrieving appropriate pictograms from large, multilingual databases, a task for which classical methods such as TF-IDF are efficient but semantically shallow, often failing when queries and pictogram descriptions diverge lexically. Despite recent progress in dense and multimodal representations, the literature still lacks a systematic comparison of textual and multimodal embeddings for the pictogram retrieval task, particularly under real usage conditions. We address this gap by comparatively evaluating three representation approaches over the ARASAAC corpus of 13,586 pictograms: TF-IDF, dense multilingual text embeddings (Sentence Transformers), and multimodal embeddings from a domain-adapted CLIP model. Our CLIP variant replaces the original English text encoder with a multilingual one and is fine-tuned on all ARASAAC pictogram–description pairs, extending linguistic coverage to Portuguese while preserving text–image alignment; this fine-tuning yields relative gains above 155% in P@1 and 167% in MRR over a zero-shot baseline. We further combine offline retrieval metrics with a user study of 34 participants spanning literal (single-concept) and compositional (multi-concept) query scenarios. The central finding is a consistent performance inversion: TF-IDF dominates literal queries (P@4 = 0.912, mean rating 4.625/5), whereas fine-tuned CLIP is markedly superior for compositional queries (P@4 = 0.806, mean rating 4.170/5), with text embeddings occupying an intermediate position in both settings. These results indicate that no single representation is universally optimal for AAC pictogram retrieval and motivate adaptive, query-aware hybrid architectures that route each query to the most suitable method.
Referências
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners.
Cabello, L., Lleida, E., Simón, J., Miguel, A., and Ortega, A. (2018). Text-to-pictogram summarization for augmentative and alternative communication. Procesamiento del Lenguaje Natural, 61:15–22.
Chang, Y.-S., Lin, C.-W., and Chen, C.-C. (2025). Ai-driven augmentative and alternative communication system for communication enhancement in cerebral palsy using american sign language recognition. Sensors and Materials, 37(9):3841–3853.
Draffan, E., Wald, M., Ding, C., and Yin, Y. (2023). Ai supporting aac pictographic symbol adaptations. Studies in health technology and informatics, 306:215–221.
Gobierno de Aragón (2024). Arasaac - aragonese center for augmentative and alternative communication. [link]. Accessed in: september 9 2025.
Leys, M. (2022). Multilingual domain-specific sentence transformers. [link]. Accessed in: 2026-01-28.
Lülf, C., Martins, D., Salles, M., Zhou, Y., and Gieseke, F. (2024). Clip-branches: Interactive fine-tuning for text-image retrieval. arXiv preprint arXiv:2406.13322.
Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.
Meta Engineering (2017). Faiss: A library for efficient similarity search. [link]. Accessed in: 2026-02-03.
Moadeli, M. (2022). Design and Evaluation of a Context Aware AAC Application for Pre-literate Children with Complex Communication Needs. PhD thesis, University of Toronto.
Pereira, J. A., Macêdo, D., Zanchettin, C., De Oliveira, A. L. I., and Fidalgo, R. D. N. (2022a). PictoBERT: Transformers for next pictogram prediction. Expert Systems with Applications, 202:117231.
Pereira, J. A., Medeiros, S. D., Zanchettin, C., and Fidalgo, R. D. N. (2022b). Pictogram Prediction in Alternative Communication Boards: a Mapping Study. In Anais do XXXIII Simpósio Brasileiro de Informática na Educação (SBIE 2022), pages 705–717, Brasil. Sociedade Brasileira de Computação - SBC.
Pereira, J. A., Pereira, J. A., Zanchettin, C., and Do Nascimento Fidalgo, R. (2024). PrAACT: Predictive Augmentative and Alternative Communication with Transformers. Expert Systems with Applications, 240:122417.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020.
Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
Robertson, S. (2004). Understanding inverse document frequency: On theoretical arguments for idf. Journal of Documentation - J DOC, 60:503–520.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models.
Salton, G. and Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5):513–523.
Schall, K., Barthel, K. U., Hezel, N., and Jung, K. (2024). Optimizing clip models for image retrieval with maintained joint-embedding alignment. arXiv preprint arXiv:2409.01936.
Shao, Y., Liu, Y., Zhang, F., Zhang, M., and Ma, S. (2019). On annotation methodologies for image search evaluation. ACM Trans. Inf. Syst., 37(3).
