domains / families / data / gguf

gguf

separate

GGUF format and llama.cpp quantization for efficient CPU/GPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.

When to use

• Deploying on consumer hardware (laptops, desktops)
• Running on Apple Silicon (M1/M2/M3) with Metal acceleration
• Need CPU inference without GPU requirements
• Want flexible quantization (Q2_K to Q8_0)
• Using local AI tools (LM Studio, Ollama, text-generation-webui)
• **Universal hardware**: CPU, Apple Silicon, NVIDIA, AMD support
• **No Python runtime**: Pure C/C++ inference
• **Flexible quantization**: 2-8 bit with various methods (K-quants)

When not to use

• Maximum accuracy with calibration on NVIDIA GPUs
• Fast calibration-free quantization for HuggingFace
• Simple integration with transformers library
• Production NVIDIA deployment with maximum speed

Skill page

Source