domains / families / data / awq

awq

separate

Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.

When to use

• Need 4-bit quantization with <5% accuracy loss
• Deploying instruction-tuned or chat models (AWQ generalizes better)
• Want ~2.5-3x inference speedup over FP16
• Using vLLM for production serving
• Have Ampere+ GPUs (A100, H100, RTX 40xx) for Marlin kernel support
• Need maximum ecosystem compatibility (more tools support GPTQ)
• Working with ExLlamaV2 backend specifically
• Have older GPUs without Marlin support

When not to use

• Tasks requiring maximum ecosystem compatibility
• Working with ExLlamaV2 backend specifically
• Using older GPUs without Marlin support
• Tasks requiring zero calibration overhead
• Fine-tuning with QLoRA
• Tasks preferring simpler integration

Skill page

Source