root-causes/

wiki/root-causes/{cause-slug}.md

One page per recurring cause. Two incidents three months apart can share a root cause — this folder is where that pattern finally shows up in one place.

What a page looks like

---
type: root-cause
name: connection pool exhaustion
status: reviewed
incidents: [2026-03-14-checkout, 2026-04-12-checkout, 2025-11-02-orders]
services:  [checkout-api, orders-api]
tags: [database, capacity, recurring]
last_updated: 2026-04-26
---

## Definition
The database connection pool saturates faster than queries drain it …

## Why it recurs
# the most important section in the wiki
All three incidents share the same pattern: a downstream call slows
down, holds connections open longer, and the pool fills up before
ops notices …

## Detection
- alert on pool_utilization > 80% (added after 2026-03-14)
- p95 latency on [[checkout-api]] climbing past 1s

## Mitigations
- 2026-03-14 — raised pool size 50 → 200 (worked for a week)
- 2026-04-12 — added auto-scale (current)
- proposed: queue-on-saturation pattern

## Related
- [[downstream-slow-call]]
- [[capacity-planning]]

Sections — in order

1. Definition — what this failure mode is, one paragraph.
2. Why it recurs — the patterns across the incidents linked above. The most valuable section in the wiki.
3. Detection — how to spot it early. Metrics, alerts, log patterns.
4. Mitigations — short-term and long-term fixes. Note which incident tried which.
5. Related — links to concepts and other root-causes.
6. Notes — open questions, contradictions.

Rules

1. File name is the cause slug in kebab-case — pick a name future-you will recognise.
2. Type-specific frontmatter: incidents, services.
3. Every incident in the frontmatter list must link back here under its Root cause section.
4. If two pages describe what is plausibly the same cause, raise it as a contradiction rather than merging silently.
5. Why it recurs beats Definition for usefulness — invest the writing time there.