Retries¶

SMG implements automatic retries with exponential backoff to handle transient failures gracefully without overwhelming recovering services.

Overview¶

Exponential Backoff¶

Space out retry attempts with increasing delays to give services time to recover.

Jitter¶

Add randomness to backoff timing to prevent thundering herd problems.

Smart Selection¶

Only retry on transient error codes that are likely to succeed on retry.

Why Retries?¶

Transient failures are common in distributed systems:

• Network timeouts: Temporary network congestion or packet loss
• Worker overload: Temporary capacity limits (429 responses)
• Intermittent errors: Brief service interruptions during deployments
• Connection issues: Worker restart or network partition

Without retries, every transient failure becomes a client-visible error. With retries, SMG handles these automatically.

Exponential Backoff with Jitter¶

SMG uses exponential backoff with jitter to space out retry attempts:

delay = initial_backoff_ms * (backoff_multiplier ^ attempt)
delay = min(delay, max_backoff_ms)
delay = delay * (1 + random(-jitter_factor, +jitter_factor))

Example Progression¶

With default settings (no jitter for clarity):

Why Jitter?¶

Without jitter, if multiple requests fail simultaneously, they all retry at exactly the same time—potentially overwhelming the recovering service. Jitter spreads out retries randomly to prevent this "thundering herd" problem.

Retryable Status Codes¶

SMG automatically retries requests that fail with these status codes:

Requests with other status codes (e.g., 400 Bad Request, 401 Unauthorized) are not retried because they would likely fail again.

Configuration¶

smg \
  --worker-urls http://w1:8000 http://w2:8000 \
  --retry-max-retries 5 \
  --retry-initial-backoff-ms 50 \
  --retry-max-backoff-ms 30000 \
  --retry-backoff-multiplier 1.5 \
  --retry-jitter-factor 0.2

Parameters¶

Recommended Configurations¶

High-Availability¶

Balanced retries for production workloads.

smg \
  --retry-max-retries 3 \
  --retry-initial-backoff-ms 100 \
  --retry-backoff-multiplier 2.0

Use when: Production APIs, multi-worker deployments

Batch Processing¶

Aggressive retries for offline workloads.

smg \
  --retry-max-retries 10 \
  --retry-initial-backoff-ms 100 \
  --retry-max-backoff-ms 60000 \
  --retry-backoff-multiplier 2.0

Use when: Batch inference, non-interactive pipelines

No Retries¶

Disable retries entirely.

smg --disable-retries

Use when: Client handles retries, testing failure scenarios

Interaction with Circuit Breakers¶

Retries and circuit breakers work together:

When a circuit is open:

• Requests are rejected immediately (no retry to that worker)
• If other healthy workers exist, the retry goes to them
• If all circuits are open, the request fails

Monitoring¶

Metrics¶

Useful PromQL Queries¶

Backoff Distribution¶

# Average backoff delay
rate(smg_worker_retry_backoff_seconds_sum[5m]) /
rate(smg_worker_retry_backoff_seconds_count[5m])

# 99th percentile backoff
histogram_quantile(0.99, smg_worker_retry_backoff_seconds_bucket)

Alert Thresholds¶

Tuning Guidelines¶

What's Next?¶

Health Checks¶

Proactive worker monitoring and failure detection.

Health Checks →

Rate Limiting¶

Protect workers from overload with token bucket rate limiting.

Rate Limiting →