BERT vs. LLM2Vec: A Comparative Study of Embedding Models for Semantic Information Retrieval - Utino, Matheus Yasuo Ribeiro; Marcacini, Ricardo Marcondes (Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), 2025)
Semantic-based Information Retrieval (IR) has significantly benefited from advances in language models and embedding techniques. This work investigates the impact of different embedding strategies on the effectiveness of semantic retrieval, using 1-NN classification and F1-score as the evaluation metric. We evaluate two model families: BERT variants and the novel LLM2Vec approach. Experiments conducted on six diverse datasets show that LLM2Vec models consistently outperform BERT-based ones across all metrics, with the Mistral-7B-Instruct-v2 model in its unsupervised configuration achieving the highest scores. Additionally, we demonstrate that LLM2Vec performance is robust to prompt variations, highlighting its practical applicability in IR systems.