domains / families / data / vllm

vllm

separate

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

When to use

• Deploying production LLM APIs (100+ req/sec)
• Serving OpenAI-compatible endpoints
• Limited GPU memory but need large models
• Multi-user applications (chatbots, assistants)
• Need low latency with high throughput
• **llama.cpp**: CPU/edge inference, single-user
• **HuggingFace transformers**: Research, prototyping, one-off generation
• **TensorRT-LLM**: NVIDIA-only, need absolute maximum performance

Skill page

Source