TIFT REDUCTION
%
Reduction in Inference Latency
The High-performance inference gateway for production LLM deployments
Route, balance, and orchestrate traffic across your LLM fleet with enterprise-grade reliability.
ROUTING LATENCY
< ms
Routing Overhead
METRICS
+
Metrics Available
OPEN AI COMPATIBLE
%
Open AI Compatible
WORKS WITH
WHY SHEPHERD MODEL GATEWAY?
SMG sits between your applications and LLM workers, providing a unified control and data plane for managing inference at scale. Whether you're running a single model or orchestrating hundreds of workers across multiple clusters, SMG gives you the tools to do it reliably.
HOW IT WORKS
Gateway Layer
Gateway
- Rate Limiter
- OIDC Auth
- WebAssembly
- Metrics
- OpenTelemetry Support
- Multi-Tenant
Router Layer
Router
- Tokenization
- Chat History
- Function Calls
- Reasoning
- MCP Execution
- Load Balancing
- Circuit Breaker
One APIAny BackendEnterprise Reliability
CHOOSE YOUR PATH
SMG sits between your applications and LLM workers, providing a unified control and data plane for managing inference at scale. Whether you're running a single model or orchestrating hundreds of workers across multiple clusters, SMG gives you the tools to do it reliably.
Start here to understand what SMG does and get it running in minutes.
New to SMGUnderstand SMG's architecture, routing strategies, and reliability features.
Learn the ConceptsContinue onboarding with monitoring, logging, and TLS guides.
Ops SetupComplete reference for the OpenAI-compatible API and SMG extensions.
API Reference