← anatomy · knowledge · incidents/
03 · KNOWLEDGE

incidents/

wiki/incidents/{YYYY-MM-DD}-{kebab-slug}.md

One page per real event. The most concrete pages in the wiki — what happened, when, who was affected, how it ended. Named with the start date in UTC.

2026-03-14-checkout SEV2 47 minutes reviewed Summary Timeline Root cause → [[connection-pool-exhaustion]] Action items [ ] payments — raise pool 50→200 — 2026-03-21

One markdown page per incident. Frontmatter on top; seven sections below.

What a page looks like

wiki/incidents/2026-03-14-checkout.mdspecimen · trimmed
---
type: incident
name: 2026-03-14 checkout latency spike
date: 2026-03-14
severity: SEV2
duration_minutes: 47
status: reviewed
owner: payments-team
services: [checkout-api, payments-db]
root_causes: [connection-pool-exhaustion]
sources: [raw/postmortems/2026-03-14-checkout.md]
tags: [latency, database, capacity]
last_updated: 2026-04-26
---

## Summary
Checkout p95 latency rose from 380ms to 12s for ~47 minutes …

## Timeline
- 14:02 — Datadog alert (checkout-p95 > 1s) pages payments oncall
- 14:11 — pool saturation confirmed on [[payments-db]]
- 14:49 — pool size raised 50 → 200; latency recovers

## Root cause
Linked to [[connection-pool-exhaustion]]; pool capacity …

## Action items
- [ ] payments-team — auto-scale pool by p95 — 2026-04-30
- [x] sre — alert on pool utilization > 80% — 2026-03-20

Sections — in order

  1. Summary — one paragraph: what happened, who was affected, how it ended.
  2. Timeline — bullet list, HH:MM — event, in UTC.
  3. Detection — how it was noticed (alert, customer report, self-reported).
  4. Root cause — link to a root-causes/ page; brief description specific to this incident.
  5. Resolution — what mitigated, what fixed.
  6. Action items — - [ ] owner — task — due. Completed items stay, struck through.
  7. Notes — open questions, contradictions, links to related incidents.

Rules

  1. File name is {YYYY-MM-DD}-{kebab-slug}.md. Date = start of incident in UTC.
  2. Type-specific frontmatter: date, severity, duration_minutes, services, root_causes.
  3. Link every named service, root-cause, and owner — first mention is a wiki-link.
  4. If a referenced entity has no page, create a stub rather than leave a dangling link.
  5. Unknown owner? Write owner: unknown and flag it under Notes.