domains / families / data / bitsandbytes

bitsandbytes

separate

Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.

When to use

• GPU memory limited (need to fit larger model)
• Training with QLoRA (fine-tune 70B on single GPU)
• Inference only (50-75% memory reduction)
• Using HuggingFace Transformers
• Acceptable 0-2% accuracy degradation
• **GPTQ/AWQ**: Production serving (faster inference than bitsandbytes)
• **GGUF**: CPU inference (llama.cpp)
• **FP8**: H100 GPUs (hardware FP8 faster)

Skill page

Source