High Availability¶

SMG supports high-availability cluster deployments using mesh networking for fault tolerance, scalability, and zero-downtime updates.

Overview¶

Scalability¶

Distribute load across multiple router instances. Add nodes without downtime.

State Synchronization¶

Share worker states, policy configurations, and rate limits across the cluster in real-time.

Zero Downtime Updates¶

Perform rolling updates without service interruption. Graceful shutdown with request draining.

Mesh Architecture¶

Cluster Coordination¶

Node coordination for cluster operations.

• Membership tracking
• Node status management
• Graceful shutdown coordination

CRDT Stores¶

Conflict-free Replicated Data Types for eventual consistency.

• No coordination locks
• Partition tolerant
• Automatic conflict resolution

State Replication¶

Real-time synchronization of all cluster state.

• Worker registry
• Rate limit counters
• Cache-aware routing trees

Configuration¶

Quick Start¶

Enable mesh networking with minimal flags:

# Start first node
smg --enable-mesh --mesh-host 0.0.0.0 --mesh-advertise-host 10.0.0.1 --mesh-port 39527

# Start second node, joining the first
smg --enable-mesh --mesh-host 0.0.0.0 --mesh-advertise-host 10.0.0.2 --mesh-port 39528 --mesh-peer-urls 10.0.0.1:39527

Command Line Options¶

Python Entrypoint¶

--enable-mesh is also available in the Python entrypoint used by the Docker image. When --mesh-host is left at 0.0.0.0, set --mesh-advertise-host to a routable address such as the pod IP:

smg launch --enable-mesh --mesh-host 0.0.0.0 --mesh-advertise-host 10.0.0.11 --mesh-port 39527

Basic Configuration¶

Node 2 (Join)

smg --enable-mesh \
    --mesh-server-name node2 \
    --mesh-host 0.0.0.0 \
    --mesh-advertise-host 10.0.0.12 \
    --mesh-port 39527 \
    --mesh-peer-urls "10.0.0.11:39527" \
    --host 0.0.0.0 \
    --port 8000

Node 3 (Join)

smg --enable-mesh \
    --mesh-server-name node3 \
    --mesh-host 0.0.0.0 \
    --mesh-advertise-host 10.0.0.13 \
    --mesh-port 39527 \
    --mesh-peer-urls 10.0.0.11:39527 \
    --host 0.0.0.0 \
    --port 8000

Gossip Protocol¶

State Synchronization¶

SMG uses a SWIM-based gossip protocol for cluster membership and state propagation:

1. Ping/Ping-Req: Each node periodically pings random peers to check health
2. State Sync: Healthy nodes exchange state information during pings
3. Failure Detection: Unreachable nodes are marked as suspected, then down
4. Broadcast: Status changes are broadcast to all cluster members

Node Status States¶

Failure Detection Timing¶

State Synchronization¶

Synchronized State Types¶

Rate Limits¶

Cluster-wide rate limiting coordination.

• Token bucket state
• Request counters
• Quota synchronization

Routing Trees¶

Cache-aware routing state shared across nodes.

• Radix tree operations
• Prefix match data
• LRU eviction coordination

Policy State¶

Routing policy configuration and state.

• Policy parameters
• Load balancing weights
• Session affinity mappings

Cache-Aware State Sync¶

Cache-aware routing policy state is synchronized across mesh nodes. This ensures that KV cache routing decisions are consistent across all routers in the cluster, preventing redundant cache misses and enabling optimal prefix reuse regardless of which router handles the request.

CRDT Implementation¶

SMG uses several CRDT types for conflict-free synchronization:

Deployment Patterns¶

Three-Node Cluster (Minimum HA)¶

Configuration

# All nodes — point at one reachable peer for bootstrap
smg --enable-mesh \
    --mesh-peer-urls node1:39527 \
    --worker-urls http://worker1:8000 http://worker2:8000

Five-Node Cluster (Higher Availability)¶

Configuration

# All nodes — point at one reachable peer for bootstrap
smg --enable-mesh \
    --mesh-peer-urls node1:39527 \
    --worker-urls http://worker1:8000 http://worker2:8000

Kubernetes Deployment¶

StatefulSet Configuration¶

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: smg
spec:
  serviceName: smg-mesh
  replicas: 3
  selector:
    matchLabels:
      app: smg
  template:
    metadata:
      labels:
        app: smg
    spec:
      containers:
      - name: smg
        image: ghcr.io/lightseekorg/smg:latest
        args:
        - --enable-mesh
        - --mesh-server-name=$(POD_NAME)
        - --mesh-host=0.0.0.0
        - --mesh-advertise-host=$(POD_IP)
        - --mesh-port=39527
        - --mesh-peer-urls=smg-0.smg-mesh:39527
        - --worker-urls=$(WORKER_URLS)
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 8000
          name: http
        - containerPort: 39527
          name: mesh

Headless Service¶

apiVersion: v1
kind: Service
metadata:
  name: smg-mesh
spec:
  clusterIP: None
  selector:
    app: smg
  ports:
  - port: 39527
    name: mesh

Kubernetes Pod Discovery¶

Use --router-selector to enable automatic pod discovery via the Kubernetes API. SMG will find and join other router pods matching the given label selector, removing the need for static --mesh-peer-urls:

smg --enable-mesh --service-discovery --router-selector app=smg tier=router

HA Management API¶

Health Endpoints¶

Cluster Status Response¶

{
  "node_name": "node1",
  "node_count": 3,
  "nodes": [
    {"name": "node1", "address": "node1:39527", "status": "1", "version": 1},
    {"name": "node2", "address": "node2:39527", "status": "1", "version": 1},
    {"name": "node3", "address": "node3:39527", "status": "1", "version": 1}
  ],
  "stores": {
    "membership_count": 3,
    "worker_count": 0,
    "policy_count": 0,
    "app_count": 0
  }
}

The status field is emitted as the stringified discriminant of the NodeStatus prost enum: "0" INIT, "1" ALIVE, "2" SUSPECTED, "3" DOWN, "4" LEAVING.

Monitoring¶

Mesh Metrics¶

Alerting Rules¶

groups:
- name: smg-mesh
  rules:
  # router_mesh_peer_connections is a per-peer gauge (0/1, labeled by "peer").
  # count(router_mesh_peer_connections == 1) gives the number of active peer links.
  # Adjust the threshold to match your expected peer count (e.g. N-1 for an N-node cluster).
  - alert: SMGClusterDegraded
    expr: count(router_mesh_peer_connections == 1) < 2
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "SMG cluster has fewer than expected peer connections"

  - alert: SMGNodeDown
    expr: router_mesh_peer_connections == 0
    for: 30s
    labels:
      severity: critical
    annotations:
      summary: "SMG mesh node {{ $labels.peer }} is down"

Best Practices¶

Availability Zones¶

Distribute nodes across availability zones for resilience against zone failures.

Network Latency¶

Keep mesh nodes in the same region (< 10ms RTT) for optimal state sync performance.

Monitoring¶

Monitor count(router_mesh_peer_connections == 1) to track active peer links and alert when the count drops below your expected threshold.

Troubleshooting¶

Common Issues¶

Debug Logging¶

RUST_LOG=smg::mesh=debug smg --enable-mesh ...

Verify Cluster Health¶

# Check cluster status
curl http://node1:8000/ha/status | jq

# Check individual node health
curl http://node1:8000/ha/health | jq

# Check worker states
curl http://node1:8000/ha/workers | jq

# Check policy states
curl http://node1:8000/ha/policies | jq

What's Next?¶

Circuit Breakers¶

Isolate failing workers to prevent cascade failures.

Circuit Breakers →

Metrics Reference¶

Complete list of mesh networking metrics.

Metrics Reference →