domains / families / devops / llm-inference-scaling

llm-inference-scaling

authentic

Auto-scale LLM inference clusters on Kubernetes using KEDA, custom GPU metrics, and horizontal pod autoscaling. Handle traffic spikes, implement queue-based scaling, and optimize cost with spot instances for AI workloads.

When to use

• LLM API traffic is unpredictable and you need to scale up/down automatically
• Managing a fleet of vLLM or TGI inference pods on Kubernetes
• Reducing inference costs with spot/preemptible GPU instances
• Implementing queue-based autoscaling for batch inference jobs
• Building a multi-model serving platform that shares GPU resources

Skill page

Source