Every serious LLM application has become a multi-model application. Teams no longer bet on one provider — they route traffic across models and vendors to optimize cost, latency, and quality. That has turned the LLM gateway into core infrastructure, the same way API gateways became core to microservices a decade ago.
This comparison ranks the top 10 LLM traffic routing tools and AI gateways of 2026 — with live star data verified August 15, 2026. NVIDIA's Switchyard (+1,195⭐/week) is the fastest-growing new entrant.
| # | Tool | ⭐ Stars | Type | Best For |
|---|---|---|---|---|
| 1 | BerriAI/litellm | 56,372 | LLM Proxy / Router | The standard self-hosted proxy — 100+ providers, one OpenAI-compatible API, budgets & fallbacks |
| 2 | Kong/kong | 43,984 | API + AI Gateway | The classic API gateway adding LLM plugins — enterprise-grade auth, rate limiting, and AI routing |
| 3 | songquanpeng/one-api | 36,383 | Unified LLM API | Self-hosted unified API with key management, channels, and quotas — huge in Asia |
| 4 | Portkey-AI/gateway | 12,724 | AI Gateway + Guardrails | OpenAI-compatible gateway with guardrails, caching, and 250+ model support |
| 5 | higress-group/higress | 9,102 | AI-Native API Gateway | Envoy-based gateway with first-class LLM routing, fallback, and key management |
| 6 | helicone/helicone | 6,072 | Observability + Routing | Log, evaluate, and route — the observability-first gateway |
| 7 | envoyproxy/ai-gateway | 1,927 | K8s-Native AI Gateway | Envoy's AI gateway for Kubernetes — the CNCF path to LLM routing |
| 8 | NVIDIA-NeMo/Switchyard | 1,492 | LLM Traffic Router | Rust router preserving native OpenAI/Anthropic API compatibility — NVIDIA's new bet |
| 9 | OpenRouter | SaaS | Managed Model Hub | 300+ models behind one API with unified billing — the managed alternative |
| 10 | awesome-ai-gateway | 85 | Category Index | 160+ gateway tools cataloged — the map of this exploding category |
NVIDIA-NeMo/Switchyard (1,492⭐, +1,195/week) is the freshest proof that LLM traffic routing is becoming its own layer. Written in Rust, it routes requests across models and providers while preserving native OpenAI and Anthropic API compatibility — no SDK rewrites. The pitch is exactly what the microservices era taught us: a gateway decouples consumers from providers, enabling benchmarking, canary rollouts, and cost/performance optimization.
✔ Native API compatibility = zero client changes; Rust = low latency and small footprint
✖ New project: enterprise features (auth, observability) are still maturing
The API-gateway incumbents are absorbing LLM routing. Kong (43,984⭐) ships AI plugins on its battle-tested gateway. Higress (9,102⭐) is Envoy-based with first-class LLM fallback and key management — the Alibaba-origin project that went community-owned. Envoy AI Gateway (1,927⭐) is the CNCF-native option for Kubernetes shops. Enterprises don't want a new box; they want their existing gateway to speak LLM.
✔ Existing control planes, auth, and rate limiting extend to LLM traffic
✖ General-purpose gateways add LLM-specific features slower than dedicated tools
LiteLLM (56,372⭐) remains the default self-hosted proxy: 100+ providers behind one OpenAI-compatible API, with budgets, fallbacks, and load balancing. One-API (36,383⭐) owns the self-hosted key-management niche (channels, quotas, user tiers). Portkey (12,724⭐) bundles guardrails and caching. Helicone (6,072⭐) leads the observability angle. The proxy layer is crowded — differentiation is now about guardrails, cost analytics, and multi-provider fallbacks.
✔ One codebase, any model — swap providers without touching application code
✖ Proxies add a hop; routing decisions need good telemetry to be trusted
| Use Case | Best Tool | ⭐ | Why |
|---|---|---|---|
| Self-hosted universal proxy | LiteLLM | 56,372 | 100+ providers, budgets, fallbacks |
| Zero-client-change routing | Switchyard | 1,492 | Native OpenAI/Anthropic compat in Rust |
| Enterprise API + AI gateway | Kong | 43,984 | Battle-tested control plane + AI plugins |
| Kubernetes-native | Envoy AI Gateway | 1,927 | CNCF path, K8s CRDs |
| Key management + quotas | One-API | 36,383 | Channels, user tiers, self-hosted |
| Managed, zero-ops | OpenRouter | SaaS | 300+ models, unified billing |
Run LiteLLM or Switchyard on a small VPS in front of your models. Deploy on DigitalOcean with a managed database for usage tracking.
DigitalOcean →Put your gateway behind Cloudflare for DDoS protection, caching, and global edge distribution — free tier handles most traffic.
Cloudflare →Gateways are critical infrastructure — monitor uptime and latency with UptimeRobot and Better Stack.
UptimeRobot → Better Stack →Disclosure: some links above are affiliate links (we may earn a commission at no extra cost to you).
| Revenue Path | Potential | Why |
|---|---|---|
| Cloud infrastructure affiliate | HIGH | Every gateway needs a host — DigitalOcean/Vercel/Cloudflare all fit |
| Observability affiliate | HIGH | Gateway telemetry is the #1 adjacent purchase — Better Stack/Datadog/UptimeRobot |
| Managed gateway SaaS affiliate | MEDIUM | Teams that don't self-host buy OpenRouter-class services |