Not All 4-bit Quantizers Are Equal: Deployment-Time Mitigation of PII Leakage in Fine-Tuned Small Language Models - Kapelinski, Cristhian; Kreutz, Diego (Anais do Simpósio Brasileiro de Cibersegurança (SBSeg); 2026: Anais do XXVI Simpósio Brasileiro de Cibersegurança, 2026)
Organizations fine-tune small language models on private data and then compress them to 4 bits for resource-efficient deployment. We show that which 4-bit method they use also affects privacy, and that what separates the methods is not the bit width but whether they tune their rounding on a small sample of text, the calibration corpus. On our primary model, when each planted record’s own opening text is used as the prompt, the two calibration-based methods we test, Activation-aware Weight Quantization (AWQ) and Gradient-based Post-Training Quantization (GPTQ), each reproduce none of the planted records, while the calibration-corpus-free GGUF Q4 K M format reproduces 5.3% of them on the same seeds. Tracked across five open models with 0.5–7 billion parameters, AWQ leaks least at every size and in both families, with little accuracy loss at 3–7 billion. Controlled experiments associate the difference with calibration-induced rounding error in channels involved in rare-token prediction. Choosing the 4-bit method is therefore a deployment-time privacy decision, not only a question of speed and quality.