domains / families / data / hqq

hqq

separate

Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4/3/2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.

When to use

• Quantizing models without calibration data (no dataset needed)
• Need fast quantization (minutes vs hours for GPTQ/AWQ)
• Deploying with vLLM or HuggingFace Transformers
• Fine-tuning quantized models with LoRA/PEFT
• Experimenting with extreme quantization (2-bit, 1-bit)
• **No calibration**: Quantize any model instantly without sample data
• **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference
• **Flexible precision**: 8/4/3/2/1-bit with configurable group sizes

When not to use

• Calibration-based accuracy requirements
• Production serving requiring AWQ
• Maximum accuracy with calibration data available (GPTQ)
• Simple 8-bit/4-bit without custom backends (bitsandbytes)
• CPU inference or Apple Silicon deployment (llama.cpp/GGUF)

Skill page

Source