A distributed system is melting down. Read the incident report.
Study the architecture: node capacities, edge types, timeouts.
Predict which node fails first — and what kills it.
WoMbat
INCIDENT
ARCHITECTURE
SYNC ASYNC stressor target partitioned cache node
DIAGNOSIS
Which node fails first, and why?
— click a node —
RESULT
HOW TO READ THE DIAGRAM
NODE STATS
conc: N
Concurrency slots — how many requests this node can handle
simultaneously. When all slots are busy, new arrivals queue
up.
q: N
Queue limit — how many requests wait while all slots are
occupied. Once full, further arrivals are dropped immediately
→ Queue Drop.
latency: N ticks
How long each request spends processing inside this node
before being forwarded downstream.
quota: N +R/t
Token bucket rate limit. Holds at most N tokens, refills R per
tick. An empty bucket means instant rejection →
Rate Limited.
hit: N%
Cache hit rate (cylinder-shaped nodes). Requests served from
memory at this rate — only the remainder reach the downstream
database. A cache flush drops this to 0%, sending every
request straight to the DB → thundering herd.
⚡ CB
Circuit breaker. After a threshold of SYNC timeout failures
within an observation window the breaker trips, fast-failing
every subsequent request for a cooldown period →
Circuit Open. After the cooldown the breaker
resets and lets one request through (half-open).
EDGES
SYNC
The caller holds its concurrency slot until the downstream
responds. A slow or unreachable downstream directly starves
the caller's capacity.
ASYNC
The caller releases its slot immediately after forwarding.
Downstream slowness doesn't back up the caller.
timeout: N
How long a SYNC caller waits before declaring the call failed
and releasing its slot.
PARTITIONED
The link silently drops all packets. The caller can't tell
this from an infinitely slow response — it waits the full
timeout on every call.
SYSTEM POLICY
retries: N (exp. backoff)
On SYNC timeout the caller retries, doubling the wait each
time (T → 2T → 4T…). The slot is held throughout. Budget
exhausted → Deadline Exceeded.
retries: none
A SYNC timeout simply releases the slot. The system limps
along at reduced throughput until a queue fills →
Queue Drop.