domains / families / devops / llm-gateway

llm-gateway

authentic

Deploy an API gateway for LLM traffic with load balancing, rate limiting, key management, semantic caching, fallback routing, and cost tracking. Covers LiteLLM Proxy, OpenRouter-compatible setup, and custom Nginx/Traefik patterns.

When to use

• Running multiple LLM backends (OpenAI, Anthropic, vLLM, Ollama) behind a single endpoint
• Enforcing per-team or per-user rate limits and spend budgets
• Implementing automatic fallback when a provider is down
• Adding semantic caching to reduce API costs by 20–50%
• Centralizing API key management instead of distributing keys to every app

Skill page

Source