Bootstrapping Text Anomaly Detection with LLM-Generated Weak Supervision

• Fabio Masaracchia Maia USP
• Anna Helena Reali Costa USP

DOI: https://doi.org/10.5753/stil.2026.26601

Resumo

Text anomaly detection is challenging because anomalous instances often share vocabulary and surface form with normal data, making them hard to distinguish without semantic understanding. Semi-supervised methods can significantly outperform unsupervised baselines, but rely on labeled anomalies that are rarely available in practice. LLMs encode rich semantic knowledge that can approximate human judgments, yet using them directly as detectors is costly at inference time and sensitive to prompt design, while generating synthetic outliers risks distribution mismatch with real anomalies. We propose a different strategy: treating a compact, locally deployed LLM as a noisy annotator over real data. The LLM scores a small subset of unlabeled documents once at training time, producing weak labels that refine a sentence encoder via contrastive fine-tuning and train a lightweight downstream detector — without cloud APIs, human annotation, or curated anomaly datasets. Across four datasets spanning two languages and two tasks, the approach recovers up to 79% of the gap to oracle upper bounds while using 14–91× fewer labeled anomalies. We further identify three failure modes that explain when LLM-generated weak supervision succeeds or breaks down.

Referências

Ait-Saada, M. and Nadif, M. (2023). Unsupervised anomaly detection in multi-topic short-text corpora. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL).

Boutalbi, K., Loukil, F., Verjus, H., Telisson, D., and Salamatian, K. (2023). Machine learning for text anomaly detection: A systematic review. In 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC), pages 1319–1324.

Breunig, M. M., Kriegel, H.-P., Ng, R. T., and Sander, J. (2000). Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, SIGMOD ’00, page 93–104, New York, NY, USA. Association for Computing Machinery.

Chandola, V., Banerjee, A., and Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3):71–97.

Das, A. S., Ajay, A., Saha, S., and Bhuyan, M. (2023). Few-shot anomaly detection in text with deviation learning. In Neural Information Processing: 30th International Conference, ICONIP 2023, Changsha, China, November 20–23, 2023, Proceedings, Part II, page 425–438, Berlin, Heidelberg. Springer-Verlag.

Garcia, K., Shiguihara, P., and Berton, L. (2024). Breaking news: Unveiling a new dataset for portuguese news classification and comparative analysis of approaches. PLOS ONE, 19(1):1–15.

Gerganov, G. et al. (2023). llama.cpp: LLM inference in C/C++. [link].

Goldstein, M. and Dengel, A. (2012). Histogram-based outlier score (HBOS): A fast unsupervised anomaly detection algorithm. In Wölfl, S., editor, KI-2012: Poster and Demo Track, pages 59–63, Saarbrücken, Germany.

He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Lin, C., Jiao, J., Yiu, S. M., Duan, N., and Chen, W. (2024). AnnoLLM: Making large language models to be better crowdsourced annotators. In Yang, Y., Davani, A., Sil, A., and Kumar, A., editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 165–190, Mexico City, Mexico. Association for Computational Linguistics.

Hendrycks, D., Mazeika, M., and Dietterich, T. (2019). Deep anomaly detection with outlier exposure. In Proceedings of the International Conference on Learning Representations (ICLR).

Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. (2020). Supervised contrastive learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 18661–18673. Curran Associates, Inc.

Lang, K. (1995). Newsweeder: Learning to filter netnews. [link]. CMU, unpublished manuscript.

Leite, J. A., Silva, D. F., Bontcheva, K., and Scarton, C. (2020). Toxic language detection in social media for brazilian portuguese: New dataset and multilingual analysis. CoRR, abs/2010.04543.

Liu, B., Zhan, L.-M., Lu, Z., Feng, Y., Xue, L., and Wu, X.-M. (2024). How good are LLMs at out-of-distribution detection? In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N., editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRECCOLING 2024), pages 8211–8222, Torino, Italia. ELRA and ICCL.

Liu, F. T., Ting, K. M., and Zhou, Z.-H. (2008). Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining, pages 413–422.

Maia, F. and Costa, A. (2025). Learning with few: A comparative study of multilingual text anomaly detection. In Anais do XVI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, pages 233–246, Porto Alegre, RS, Brasil. SBC.

Manolache, A., Brad, F., and Burceanu, E. (2021). DATE: Detecting anomalies in text via self-supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL).

Pang, G., Shen, C., Cao, L., and Hengel, A. V. D. (2021). Deep learning for anomaly detection: A review. ACM Comput. Surv., 54(2).

Qwen Team (2025). Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., and Ré, C. (2020). Snorkel: rapid training data creation with weak supervision. The VLDB Journal, 29(2-3):709–730.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Inui, K., Jiang, J., Ng, V., and Wan, X., editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S. A., Binder, A., Müller, E., and Kloft, M. (2018). Deep one-class classification. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4393–4402. PMLR.

Ruff, L., Vandermeulen, R. A., Görnitz, N., Binder, A., Müller, E., and Kloft, M. (2020). Deep semi-supervised anomaly detection. In International Conference on Learning Representations (ICLR).

Ruff, L., Zemlyanskiy, Y., Vandermeulen, R., Schnake, T., and Kloft, M. (2019). Self-attentive, multi-context one-class classification for unsupervised anomaly detection on text. In Korhonen, A., Traum, D., and Màrquez, L., editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4061–4071, Florence, Italy. Association for Computational Linguistics.

Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., and Williamson, R. C. (2001). Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443–1471.

Sener, O. and Savarese, S. (2018). Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations (ICLR).

Sharma, R. (2018). Twitter sentiment analysis for hate speech detection. [link]. Accessed: 2025-09-03.

Tunstall, L., Reimers, N., Jo, U. E. S., Bates, L., Korat, D., Wasserblat, M., and Pereg, O. (2022). Efficient few-shot learning without prompts. Presented at the NeurIPS 2022 Workshop on Efficient Natural Language and Speech Processing (ENLSP-II).

Wang, S., Liu, Y., Xu, Y., Zhu, C., and Zeng, M. (2021). Want to reduce labeling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Xu, A., Ren, X., and Jia, R. (2023a). Contrastive novelty-augmented learning: Anticipating outliers with large language models. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11778–11801, Toronto, Canada. Association for Computational Linguistics.

Xu, H. (2022). Deepod: A benchmarking framework for deep outlier detection. [link].

Xu, R. and Ding, K. (2025). Large language models for anomaly and out-of-distribution detection: A survey. In Chiruzzo, L., Ritter, A., and Wang, L., editors, Findings of the Association for Computational Linguistics: NAACL 2025, pages 6007–6027, Albuquerque, New Mexico. Association for Computational Linguistics.

Xu, Y., Gábor, K., Milleret, J., and Segond, F. (2023b). Comparative analysis of anomaly detection algorithms in text data. In Mitkov, R. and Angelova, G., editors, Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 1234–1245, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.

Yang, T., Nian, Y., Li, L., Xu, R., Li, Y., Li, J., Xiao, Z., Hu, X., Rossi, R. A., Ding, K., Hu, X., and Zhao, Y. (2025). AD-LLM: Benchmarking large language models for anomaly detection. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T., editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 1524–1547, Vienna, Austria. Association for Computational Linguistics.

Zhao, Y., Nasrullah, Z., and Li, Z. (2019). Pyod: A python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20(96):1–7.

Zhou, Z.-H. (2018). A brief introduction to weakly supervised learning. National Science Review, 5(1):44–53.