Elevated 5xx at the edge (2026-09-15)

Summary

For 34 minutes on 15 September 2026, one of the two edge-gateway replicas returned intermittent 5xx responses. The trigger was unremarkable: the Redis primary failed over during a routine upgrade, as it is designed to. The response was not: the gateway’s Redis client reconnected in an unbounded tight loop, saturated a worker-thread pool, and made every rate-limit lookup on that replica time out until the replica was restarted. The public impact was roughly half of a half, about 9% of requests over the window, because Caddy’s round-robin load balancing kept steering every other request onto the healthy replica. Nobody’s message was lost; some requests needed a client-side retry.

What customers saw

The fleet never went dark, which made this incident quieter than SEV-2s usually look. Two failure signatures were visible from the outside:

• Intermittent 503s with a retry-after header on /v1/events, roughly every other request for the duration of the window.
• Slightly elevated p99 enqueue latency while the healthy replica absorbed the stranded traffic (the benchmark ceiling of 50ms was never breached).

Delivery was not affected. Nothing sat in a dead letter queue, nothing was lost, and no subscriber-visible behavior changed. This was an availability defect at the ingest edge, and a small one at that, but it was a defect we had shipped ourselves, in code written to survive exactly this event.

Timeline (UTC)

• 08:50: Planned Redis upgrade begins; primary fails over. Sentinel promotes a replica within 12 seconds. Nothing in this part deviated from the drill we ran six weeks ago.
• 09:41: The gateway’s Redis client begins its reconnect loop. The loop has no jitter and no cap: each failure schedules the next attempt immediately.
• 09:43: The reconnect loop saturates the replica’s blocking worker-thread pool. Rate-limit lookups and idempotency-lock reads start timing out; requests surface as 503s. The unhealthy period has begun.
• 09:52: Alert fires: Gateway (replica 2) degraded on the status page. The external synthetic monitor against api.nervly.io/v1/health stays green the whole time; Caddy keeps balancing onto the healthy replica, so the fleet-level probe never trips.
• 09:54: On-call identifies the reconnect storm from logs: tens of thousands of connection attempts in nine minutes, all against a primary that is now healthy and unfazed.
• 10:05: Hotfix prepared: bounded exponential backoff with full jitter.
• 10:15: Replica restarted with the fix. Error rate to zero. Incident closed ten minutes later after a clean soak.

Root cause

The gateway treats Redis as optional-but-fast: rate-limit counters and idempotency locks live there, and a failed lookup degrades to a permissive answer rather than failing the request. That degradation path is correct. What was broken was everything before it.

The reconnect policy was fail → immediately retry, forever. Redis failovers take seconds; a capped, jittered loop would have reconnected within five seconds and nobody would have noticed anything. Instead the tight loop held a full core spinning and, because the Go Redis client shares a connection pool whose construction contends on the same mutexes the request path needs, starved the blocking pool that serves lookups. The lookups slowed to the point where the degradation path itself exhausted: requests were queued behind reconnect attempts for longer than their own deadline, and the permissive fallback could not save them.

In short: a failure of pacing, not of architecture. Every individual design decision (in-memory degradation, separate pools, health endpoints) was sound. The missing piece was a bound on how hard we try to reconnect.

Why it took 34 minutes

Half of that window was detection and diagnosis: the external monitor was green (by design it probes the fleet, not the units), so the first signal was the per-replica status check firing at 09:52. The rest was the hotfix and restart. The post-incident fix list exists because each of those minutes had a name.

What we changed

1. Bounded reconnect backoff. Exponential backoff with full jitter, capped at 5s. The tight loop is gone; reconnection after a failover now completes within seconds, invisibly.
2. Readiness gates the subscription. A replica with a disconnected Redis reports degraded on /v1/health, and Caddy’s max_fails pulls it from rotation within seconds. A half-broken replica no longer serves half-broken traffic; it serves none.
3. Per-replica status checks. The status page checks each gateway replica individually (Gateway (replica 1/2)), so a single dead replica is visible even while the edge still answers. The fleet-level probe remains the external monitor’s job; this incident is precisely why alerts.yaml insists those are different jobs.

What went well

• No data loss anywhere: the queue semantics held under partial failure.
• The replica itself never crashed or served wrong answers: every 5xx was honest, retryable, and header-annotated.
• Detection-to-diagnosis took two minutes once the right probe fired.

Where we got lucky

The load balancer masking the incident is a double-edged property. It kept customers whole this time; it also kept the fleet-level signal blind for eleven minutes. If the same reconnect storm had hit both replicas (a coordinated failover, for instance), the masking would have been a blackout.

Action items

The drill is the important one. Redis failover is not an edge case in this system; it is a recurring operation. An incident that can only happen once per upgrade cycle is not an incident we have fixed; it is one we have not yet repeated.