Elevated 5xx at the edge (2026-09-15)

Summary

For 34 minutes on 15 September 2026, one of the two edge-gateway replicas returned intermittent 5xx responses. The trigger was unremarkable: the Redis primary failed over during a routine upgrade, as it is designed to. The response was not: the gateway’s Redis client reconnected in an unbounded tight loop, saturated a worker-thread pool, and made every rate-limit lookup on that replica time out until the replica was restarted. The public impact was roughly half of a half — about 9% of requests over the window — because Caddy’s round-robin load balancing kept steering every other request onto the healthy replica. Nobody’s message was lost; some requests needed a client-side retry.

What customers saw

The fleet never went dark, which made this incident quieter than SEV-2s usually look. Two failure signatures were visible from the outside:

• Intermittent 503s with a retry-after header on /v1/events, roughly every other request for the duration of the window.
• Slightly elevated p99 enqueue latency while the healthy replica absorbed the stranded traffic (the benchmark ceiling of 50ms was never breached).

Delivery was not affected. Nothing sat in a dead letter queue, nothing was lost, and no subscriber-visible behavior changed. This was an availability defect at the ingest edge, and a small one at that — but it was a defect we had shipped ourselves, in code written to survive exactly this event.

Timeline (UTC)

• 08:50 — Planned Redis upgrade begins; primary fails over. Sentinel promotes a replica within 12 seconds. Nothing in this part deviated from the drill we ran six weeks ago.
• 09:41 — The gateway’s Redis client begins its reconnect loop. The loop has no jitter and no cap: each failure schedules the next attempt immediately.
• 09:43 — The reconnect loop saturates the replica’s blocking worker-thread pool. Rate-limit lookups and idempotency-lock reads start timing out; requests surface as 503s. The unhealthy period has begun.
• 09:52 — Alert fires: Gateway (replica 2) degraded on the status page. The external synthetic monitor against api.nervly.io/v1/health stays green the whole time — Caddy keeps balancing onto the healthy replica, so the fleet-level probe never trips.
• 09:54 — On-call identifies the reconnect storm from logs: tens of thousands of connection attempts in nine minutes, all against a primary that is now healthy and unfazed.
• 10:05 — Hotfix prepared: bounded exponential backoff with full jitter.
• 10:15 — Replica restarted with the fix. Error rate to zero. Incident closed ten minutes later after a clean soak.

Root cause

The gateway treats Redis as optional-but-fast: rate-limit counters and idempotency locks live there, and a failed lookup degrades to a permissive answer rather than failing the request. That degradation path is correct. What was broken was everything before it.

The reconnect policy was fail → immediately retry, forever. Redis failovers take seconds; a capped, jittered loop would have reconnected within five seconds and nobody would have noticed anything. Instead the tight loop held a full core spinning and — because the Go Redis client shares a connection pool whose construction contends on the same mutexes the request path needs — starved the blocking pool that serves lookups. The lookups slowed to the point where the degradation path itself exhausted: requests were queued behind reconnect attempts for longer than their own deadline, and the permissive fallback could not save them.

In short: a failure of pacing, not of architecture. Every individual design decision — in-memory degradation, separate pools, health endpoints — was sound. The missing piece was a bound on how hard we try to reconnect.

Why it took 34 minutes

Half of that window was detection and diagnosis: the external monitor was green (by design it probes the fleet, not the units), so the first signal was the per-replica status check firing at 09:52. The rest was the hotfix and restart. The post-incident fix list exists because each of those minutes had a name.

What we changed

1. Bounded reconnect backoff. Exponential backoff with full jitter, capped at 5s. The tight loop is gone; reconnection after a failover now completes within seconds, invisibly.
2. Readiness gates the subscription. A replica with a disconnected Redis reports degraded on /v1/health, and Caddy’s max_fails pulls it from rotation within seconds. A half-broken replica no longer serves half-broken traffic; it serves none.
3. Per-replica status checks. The status page checks each gateway replica individually (Gateway (replica 1/2)), so a single dead replica is visible even while the edge still answers. The fleet-level probe remains the external monitor’s job — this incident is precisely why alerts.yaml insists those are different jobs.

What went well

• No data loss anywhere: the queue semantics held under partial failure.
• The replica itself never crashed or served wrong answers — every 5xx was honest, retryable, and header-annotated.
• Detection-to-diagnosis took two minutes once the right probe fired.

Where we got lucky

The load balancer masking the incident is a double-edged property. It kept customers whole this time; it also kept the fleet-level signal blind for eleven minutes. If the same reconnect storm had hit both replicas — a coordinated failover, for instance — the masking would have been a blackout.

Action items

The drill is the important one. Redis failover is not an edge case in this system; it is a recurring operation. An incident that can only happen once per upgrade cycle is not an incident we have fixed — it is one we have not yet repeated.