Silent failures.
The failures that answer 200 OK.
A gateway can drop a conversation turn, splice two tool calls into one malformed argument string, or quietly bill you for work it threw away, and report perfect health throughout. We catalogued the ones we could prove, and the most severe of them are invisible to every metric the industry watches.
FailureAtlas: A Taxonomy of Failure Modes in Multi-Provider LLM Serving Infrastructure. Vishal Pandey and Gopal Singh, Metriqual.
- Layers by detectability
- 5 × 2
- the taxonomy grid
- Verified catalog entries
- 5
- 2 first-hand, 3 surveyed
- Entries that are silent
- 2
- both operationally severe
- Entries in model behaviour
- 0
- the row stays empty
Two axes, because triage asks two questions.
Where in the stack did this break, and why did nobody catch it sooner. The second question is the one that decides how bad it gets.
Multi-provider gateways occupy a position that did not exist when the classical failure taxonomies were written: a reverse proxy between application code and model providers, inheriting network partitions and thundering herds from distributed systems and streaming token protocols, tool-call index tracking and conversation state from LLM serving.
A single-axis taxonomy organised by symptom fails immediately. A retry storm and a leaked Redis semaphore both surface as HTTP 429, but one needs client-side jitter and the other needs server-side cleanup. Sorting by origin layer separates them. Sorting by detectability explains why one of them was fixed in hours and the other ran for months.
A third axis for severity turned out to be redundant. Severity is very nearly determined by detectability: loud failures get fixed because monitoring catches them, and silent ones persist because nothing flags them.
Loud
5xx errors, unhandled exceptions, container crashes, hard 429s, health-check timeouts. High signal, immediate stack trace, on-call alert. Unpleasant but tractable: the incident pipeline is built for exactly this.
Silent
HTTP 200, every health check green, and the payload wrong. A dropped turn, two tool-call arguments concatenated into invalid JSON, a cheaper model substituted. Nothing that inspects status codes, latency percentiles or error rates will see it.
Five layers, and one row we could not fill.
Every entry is placed in exactly one cell by where it originates and whether anything would tell you it happened.
| Origin layer | Loud | Silent |
|---|---|---|
| L1 Network / Transport | Event-loop block | no evidence-grade entry |
| L2 Streaming / Protocol | no evidence-grade entry | Tool-call index collision |
| L3 State / Session | no evidence-grade entry | Concurrency race condition |
| L4 Model behaviour | no evidence-grade entry | no evidence-grade entry |
| L5 Governance / Cost | Redis semaphore leak, retry storm | no evidence-grade entry |
Evidence grade, or it does not go in.
Dozens of candidate reports were filtered down to five. The bar was deliberately high enough to make the catalog small.
Two entries came from our own stress-testing campaigns during ContinuityBench, where hundreds of parallel agent instances ran against a mock gateway that simulated real latency, rate limits and transient 502s. Both failures emerged under sustained load and were documented as they happened, from system logs and from drops in the semantic continuity metric.
The other three came from a targeted survey of the public issue trackers for LiteLLM, Portkey and OpenRouter, using scoped queries per layer. That search returned dozens of candidates. Most were rejected.
Validation was manual: read the whole thread, examine the logs, trace the execution path through the open-source code, and check the merged fix to confirm the described mechanism matches what the code actually did. Three of the five ship with standalone reproduction scripts. The other two depend on a live Kubernetes probe and a running Redis instance, so they ship with setup instructions instead.
Inclusion criteria
- Concrete and specific
- not “it got worse”
- Verifiable provenance
- a primary-source URL
- Mechanistically explained
- root cause, not a trace
- Catalog entries
- 5
- With reproductions
- 3
- First-hand
- 2
- Surveyed
- 3
What each one actually does.
Five entries, each with a mechanism you can follow from trigger to symptom.
| Failure | Layer | Detectability | Source |
|---|---|---|---|
| Concurrency race condition | State / Session | Silent | First-hand |
| Tool-call index collision | Streaming / Protocol | Silent | litellm#33678 |
| Redis semaphore leak | Governance / Cost | Loud | litellm#20256 |
| Sync health check blocking | Network / Transport | Loud | litellm#21033 |
| Failover retry storm | Governance / Cost | Loud | First-hand |
Two tool calls become one broken argument
Clients rebuild parallel tool-call arguments by accumulating chunks keyed to an index. A proxy that derives its own top-level index can reset the counter on every chunk, stamping all of them index=0. The client dutifully concatenates arguments from independent tools into one string and saves it to history.
Nothing fails at the time. The error arrives on the next turn, when the corrupted history is sent back and parsing dies, which is why the investigation started three layers below the actual bug. The fix is a counter that survives across chunks.
A rate limiter that never lets go
A Redis-backed semaphore counts in-flight requests. It increments on arrival and decrements on completion, unless the upstream provider times out and the error path skips the decrement. Each upstream failure strands one count.
Given enough upstream errors the counter reaches its limit with nothing in flight, and the gateway rejects every new request for that key with a 429 permanently. The fix is a TTL on the counter, or a try/finally that decrements on every path. Both governance entries in the catalog are triggered by upstream failure, which is not a coincidence: control planes are tested against the happy path and are needed most at the exact moment they are least exercised.
The two we found ourselves.
Controlled enough to isolate the mechanism, reproduce it under load, and verify the fix.
Context bleeding between tenants
At 5 concurrent sessions the proxy scored near-perfect continuity. At 100 it collapsed to roughly 28%, a 72 point drop, because a shared history cache was being mutated by parallel tasks across an await. The payload for one conversation captured an anchor set milliseconds earlier by another.
The proxy never crashed. It returned 200 for every request. No latency, error-rate or saturation metric moved. The only signal was the semantic one.
Failover that becomes the outage
When the primary returned a transient 502, all 100 agents hit it at once and all retried on the same fixed interval. They synchronised into a herd that saturated the provider's rate limit on every retry window, and 25% of requests failed permanently over a momentary upstream hiccup.
The provider bills for tokens generated before a mid-stream failure, so every retry wave was billable work for zero completed tasks. Exponential backoff with jitter de-synchronises the waves; a circuit breaker sheds load instead of queueing thousands of doomed requests.
Three mismatches between how these systems are built and how they are watched.
Web infrastructure is stateless by design. Load balancers, proxies and REST APIs assume request A has no bearing on request B. Multi-turn LLM applications are deeply stateful, and the conversation history is the primary input to the model. Gateways that manage that state internally, or reconstruct it on the fly, are fighting their own primitives, and when those workarounds fail they do not crash. They mutate the payload.
Success is defined at the transport layer. A 200 inside the latency budget counts as a success, even if the proxy dropped a critical turn and the model answered confidently from a hole in its context. The success metric is decoupled from task correctness.
And there is no convention for monitoring semantic integrity. Distributed systems have decades of monitoring practice; LLM pipelines have manual spot checks. This creates a survivorship bias in the engineering record: the bugs that get reported are disproportionately loud, because loud is what monitoring is built to detect. Silent failures are not rare. They are under-represented because the infrastructure for noticing them does not exist yet in most deployments.
What this catalog is not.
Five entries is a starting vocabulary, not a survey of the failure space.
The catalog is not exhaustive, and the empty model behaviour row is the clearest admission of that: anecdotal reports of drift are everywhere, and rigorous documentation of it is nowhere.
Evidence quality varies. The two first-hand entries have controlled experimental data and custom reproduction harnesses. The surveyed entries rest on public issue trackers, some documented by many users and some by a single detailed report.
The two axes are a pragmatic choice aimed at operational utility, not a provable decomposition of failure space. Someone could reasonably argue for a recovery-time axis or different layer boundaries. And two of the five entries, the most thoroughly characterised ones, were found by us, using a harness built to stress exactly state management and retry policy. That is a self-sourcing bias, and the fix for it is community-contributed entries.
Where the taxonomy came from
The race condition and the retry storm were both discovered while building ContinuityBench, our benchmark for conversational continuity under provider failover. The taxonomy exists because we could not find either failure documented anywhere, and could not detect either one with a standard monitoring tool.
The catalog takes pull requests.
Entries are structured YAML with an evidence block, a mechanism, a mitigation and, where the failure can be isolated, a runnable reproduction. New failure modes will appear faster than any one team can document them, which is the whole reason it is open.