gRPC Pipeline¶

When workers communicate via gRPC, SMG becomes a complete OpenAI-compatible server with a sophisticated request processing pipeline for reasoning extraction, tool call parsing, and MCP execution.

Overview¶

Tokenization Caching¶

Two-level tokenization cache reduces CPU overhead by 60-90% for repeated content.

Reasoning Extraction¶

Extract chain-of-thought content from thinking models (DeepSeek-R1, Qwen3, etc.).

Tool Call Parsing¶

Parse function calls and execute MCP tools with automatic result injection.

Pipeline Architecture¶

HTTP Mode¶

Gateway = Smart Proxy

SMG handles routing, load balancing, and failover. Workers run full OpenAI-compatible servers.

Responsibility Comparison¶

Reasoning Parsers¶

Reasoning parsers extract chain-of-thought content from model outputs. Essential for models that produce thinking tokens before their final response.

Configuration¶

Supported Parsers¶

Qwen3

• Pattern: *qwen3*
• Initial state: Not in reasoning
• Tokens: <think> / </think>

smg --reasoning-parser qwen3

Kimi

• Pattern: *kimi*
• Initial state: Not in reasoning
• Tokens: Unicode markers

smg --reasoning-parser kimi

GLM-4.5

• Pattern: *glm45*, *glm47*
• Initial state: Not in reasoning
• Tokens: <think> / </think>

smg --reasoning-parser glm45

Complete Parser Reference¶

Output Format¶

When separate_reasoning: true is set in the request:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "The answer is 42.",
      "reasoning_content": "Let me think step by step..."
    }
  }]
}

Tool Call Parsers¶

Tool call parsers extract function calls from model output and validate arguments against schemas.

Configuration¶

Supported Parsers¶

DeepSeek

DeepSeek V3 tool format.

<tool_call>
get_weather(location="NYC")
</tool_call>

Qwen

Qwen model JSON tool calling format.

{"name": "get_weather", "arguments": {"location": "NYC"}}

Qwen XML

Qwen3-Coder / Qwen3.5+ XML format with parameter tags.

<tool_call><function=get_weather><parameter=location>NYC</parameter></function></tool_call>

Complete Parser Reference¶

Tool Execution Flow¶

1. Parse: Extract tool calls from model output
2. Validate: Check arguments against tool schema
3. Execute: Run MCP tools or return to client
4. Inject: Add tool results back to conversation
5. Continue: Resume generation if needed

Configuration¶

Parser CLI Options¶

MCP Integration¶

When MCP is configured, tool calls can be executed automatically:

smg \
  --mcp-config-path /path/to/mcp.json \
  --tool-call-parser llama

See the MCP Guide for detailed configuration.

Recommended Configurations¶

Tool Calling Model¶

Llama with MCP tool execution.

smg \
  --model-path meta-llama/Llama-3.2-70B-Instruct \
  --tool-call-parser llama \
  --mcp-config-path /config/mcp.json

Full Pipeline¶

Complete configuration with all features.

smg \
  --model-path Qwen/Qwen2.5-72B-Instruct \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --mcp-config-path /config/mcp.json \
  --tokenizer-cache-enable-l0 \
  --tokenizer-cache-enable-l1 \
  --worker-urls grpc://worker:50051

Monitoring¶

Pipeline Metrics¶

Debug Logging¶

# Enable pipeline debug logging
RUST_LOG=smg::pipeline=debug smg ...

# Enable parser debug logging
RUST_LOG=smg::parsers=debug smg ...

Troubleshooting¶

What's Next?¶

MCP Integration¶

Configure Model Context Protocol servers for tool execution.

MCP →

Cache-Aware Routing¶

Maximize KV cache hits with prefix-based routing.

Cache-Aware Routing →