Evaluating Quantized Small Language Models in a TableRAG Pipeline for Table Question Answering

• Eduardo Da Paz Rodrigues UFRGS
• Luciana Bencke UFRGS

DOI: https://doi.org/10.5753/eniac.2026.31544

Resumo

Large Language Models (LLMs) show strong performance in Table Question Answering (TableQA), but their high computational requirements restrict local deployment. This work investigates whether TableRAG-inspired Retrieval-Augmented Generation pipelines can be adapted to quantized Small Language Models (SLMs) in constrained hardware environments. We evaluate multiple SLMs on the WikiTableQuestions (WTQ) and OTT-QA benchmarks, analyzing preprocessing strategies, prompting approaches, and lightweight retrieval mechanisms. The results show that semantic context enrichment improves Text-to-SQL generation, with the Embedding-content-aware strategy achieving the best performance. The 4-bit quantized Gemma-4-E4B (Ollama) model reached 47.9% denotation accuracy on WTQ, while Qwen2.5-Coder (7B) achieved 42.0%. For table retrieval, the proposed dual-stage approach combining bi-encoder retrieval and cross-encoder reranking improved HITS@1 from 64.2% to 70.7% and MRR from 0.73 to 0.78 on OTT-QA. These results indicate that quantized SLMs can be viable for local TableQA applications under constrained environments.

Referências

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2022). On the opportunities and risks of foundation models. ACM Computing Surveys, 55(12):1–45.

Chen, W., Chang, M.-W., Schlinger, E., Wang, W., and Cohen, W. W. (2021). Open question answering over tables and text. In International Conference on Learning Representations (ICLR).

de Souza, E. A., da Silva, P. F., Gomes, D., Batista, V., Batista, E., and Pacheco, M. (2024). TableRAG: A novel approach for augmenting LLMs with information from retrieved tables. In Anais do Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL), pages 182–191, Belém, PA, Brazil. Sociedade Brasileira de Computação.

Dettmers, T. et al. (2023). Qlora: Efficient finetuning of quantized llms. NeurIPS.

Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2023). GPTQ: Accurate post-training quantization for generative pre-trained transformers. In International Conference on Learning Representations (ICLR).

Freitas Filho, J. G. M. d. and Fernandes, S. R. (2025). Do slm agents screen papers as well as llms? In Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC).

Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. (2023). Text-to-SQL empowered by large language models: A benchmark evaluation. arXiv preprint arXiv:2308.15363.

Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. In Webber, B., Cohn, T., and He, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Lewis, V., Riedel, S., Rocktäschel, T., Yogatama, D., Welbl, D., Chang, K.-W., Goyal, N., Macdonald, R., Lewis, K., Leary, R., Fan, A., Joulin, A., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 9459–9474. Curran Associates, Inc.

Li, J., Cheng, X., Zhao, W. X., Nie, J.-Y., and Wen, J.-R. (2023). HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, Singapore. Association for Computational Linguistics.

Pasupat, P. and Liang, P. (2015). Compositional semantic parsing on semi-structured tables. In ACL.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Shi, T., Zhao, C., Boyd-Graber, J., Daumé III, H., and Lee, L. (2020). On the potential of lexico-logical alignments for semantic parsing to SQL queries. In Findings of the Association for Computational Linguistics: EMNLP 2020.

Sousa, E., Santos, E., Sampaio, A., and Bezerra, C. (2025). Comparing structural quality of code generated by llms: A static analysis of code smells. In Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC).

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Xiao, S., Liu, Z., Zhang, P., and Muennighoff, N. (2023). C-pack: Packaged resources to advance general Chinese embedding. arXiv preprint arXiv:2309.07597.

Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. (2018). Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.

Yu, X., Jian, P., and Chen, C. (2025). TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10023–10037, Miami, Florida, USA. Association for Computational Linguistics.