domains / families / data / gptq

gptq

separate

Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.

When to use

• Need to fit large models (70B+) on limited GPU memory
• Want 4× memory reduction with <2% accuracy loss
• Deploying on consumer GPUs (RTX 4090, 3090)
• Need faster inference (3-4× speedup vs FP16)
• Need slightly better accuracy (<1% loss)
• Have newer GPUs (Ampere, Ada)
• Want Marlin kernel support (2× faster on some GPUs)
• Need simple integration with transformers

When not to use

• Tasks requiring <1% accuracy loss (use AWQ)
• Newer GPUs (Ampere, Ada) needing Marlin kernel support (use AWQ)
• Simple integration with transformers (use bitsandbytes)
• 8-bit quantization (use bitsandbytes)
• Tasks not needing pre-quantized model files (use bitsandbytes)

Skill page

Source