Stateful failover.
An outage your dashboard never sees.
A provider drops for eleven seconds. The retry works, the fallback answers, every health check stays green. The user still has to start the conversation over, because the model that answered received none of it. We built a benchmark for that gap and ran it against our own gateway.
ContinuityBench: A Benchmark and Systems Study of Stateful Failover in Multi-Provider LLM Routing. Vishal Pandey and Gopal Singh, Metriqual.
- Context preserved on failover
- 99.20%
- 744 of 750 events
- Preserved by stateless failover
- 0.00%
- 0 of 750 events
- Mean added latency
- +59ms
- median −450ms
- Injected failures per system
- 750
- 5 runs × 150 conversations
Availability and continuity are not the same property.
Gateways are built to guarantee that a response comes back. None of them guarantee the response knows what was said before it.
Multi-provider gateways such as LiteLLM, Portkey and OpenRouter route at the request level. Each completion is an independent transaction, so when the primary provider fails, the same payload is forwarded elsewhere. That payload is the current turn. The dialogue that preceded it lived with the provider that just went away.
In a voice agent the loss reads as a hang-up: the caller is asked to repeat themselves. In an agent pipeline it is worse, because nothing visibly breaks. The agent keeps working from a truncated context and produces plausible, wrong output several steps later, long after the state was actually lost.
The paper defines that gap as a measurable property and gives it two metrics, then tests whether a proxy can close it.
Continuity Preservation Rate
The share of failover events where the fallback provider's answer shows access to a fact established before the failure. Reported with a 95% Wilson interval, which stays honest at rates near 0 and 1.
CPR = (1/N) Σ preserved(f_i)
Continuity Latency Overhead
The paired per-conversation latency difference between forwarding the history and discarding it. Reported as mean, median and P95, because the tail is where forwarding a large payload would show up.
CLO = (1/N) Σ (ℓ_treat − ℓ_base)
Two systems, one line of difference.
Both proxies share the interceptor, the fault injector, the provider adapters and the logging. They diverge only where the failover payload is built.
last_msg = extract_last_user_message(messages)
failover_payload = [{"role": "user", "content": last_msg}]failover_payload = messages # forwarded in full
The test suite is 150 synthetic conversations of 7 to 11 turns. Turn 0 plants a verifiable anchor. The turns after it are unrelated filler that push the anchor away from the question. The final user turn probes for it, and can only be answered by a model that received the earlier turns. Anchors are split 30 apiece across five types: a stated preference, an invented term, a proper name, a numeric value and a date.
Failures are not left to chance. A manifest generated from seed 42 assigns every conversation a failure turn, a failure mode and a fallback provider, and both proxies read the same manifest. The failure fires on the probe turn itself, which is the hardest moment to fail: the model has to answer the context-dependent question from whatever context it is handed.
Primary is gpt-4o-mini, fallback is claude-3-5-sonnet, both at temperature 0. Crossing model families is deliberate. Claude has no other source for a name invented by the user four turns earlier, so a correct answer proves the history arrived.
Scoring runs through a GPT-4o judge that sees only the expected fact, the probe and the answer, never the conversation. On a 20-example calibration set covering paraphrases, hallucinated substitutions, refusals and error responses, it agreed with hand labels 19 times out of 20. The bar to proceed was 90%.
Run configuration
- Conversations
- 150 per run
- Runs
- 5 independent
- Failover events
- 750 per system
- Concurrency
- 100 simultaneous
- Failure modes
- timeout, 502, 429
- Retry policy
- min(30s, 2^a + U(0,1))
- Max retries
- 5 per failover
- Manual audits
- 15 early, 10 final
744 of 750 conversations survived the switch.
The stateless baseline preserved context in none of them, which is what the architecture predicts: it never sends the history in the first place.
| Turns | n | Baseline | Treatment |
|---|---|---|---|
| 7 turns | 44 | 0.0% | 97.7% |
| 9 turns | 58 | 0.0% | 100.0% |
| 11 turns | 48 | 0.0% | 100.0% |
| Milliseconds | Mean | Median | P25 | P75 | P95 |
|---|---|---|---|---|---|
| Baseline latency | 6,225 | 4,311 | 3,123 | 7,313 | 18,225 |
| Treatment latency | 6,283 | 4,457 | 2,600 | 6,729 | 17,584 |
| Overhead, paired | +59 | −450 | −1,687 | +1,295 | +13,614 |
The mechanism is obvious. Running it at 100 concurrent sessions is not.
Both of these took down a working prototype during stress testing. Neither appears in the gateway literature.
Conversations bled into each other
At 5 concurrent sessions the proxy scored near perfect. At 100 it collapsed to about 28%. The fallback was receiving message arrays that contained fragments of other people's conversations: a shared history cache was being mutated by parallel tasks between yields, so a payload bound for one conversation captured an anchor set milliseconds earlier by another.
The fix was deep-copying each message array before any asynchronous yield. The warning generalises: any gateway that caches conversations internally to keep client payloads small needs per-session locking, or it will leak context across tenant boundaries at exactly the moment a failover happens.
The retry loop became the outage
Testing Gemini as a fallback crashed the proxy with connection errors and drove the preservation rate to zero. The fallback allowed 15 requests per minute. When 100 sessions failed over at once it rate-limited most of them, they all slept exactly 2.0 seconds, and they all woke together. The rate-limit window never got a chance to reset, and sustained retries exhausted local sockets.
Exponential backoff with jitter, min(30s, 2^a + U(0,1)), desynchronises the threads and was enough on its own. The 750-event run completed afterwards without a single dropped connection. A retry loop tuned against an elastic primary behaves like a denial of service against a rate-limited secondary.
What the number does not cover.
The benchmark proves the mechanism is correct. It is not a production latency measurement, and we do not present it as one.
The conversations are synthetic. That is what makes the ground truth verifiable, and it also means real users, who refer back to earlier turns far more obliquely than a direct probe does, are a harder test. Validating the rate on annotated production logs is still to do.
The judge is a model. It agreed with hand labels 19 times out of 20 on calibration and the manual audit of the final run found no false positives in 10 sampled cases, but automated scoring cannot fully rule out a judge that penalises correct, unexpected phrasing.
One provider chain was measured, OpenAI to Anthropic. Gemini was cut because its free-tier limit was incompatible with the concurrency level. Evaluation is text-only and batch, so streaming failover, where a turn has to be reconstructed mid-stream, is untested. Each conversation fails once; rolling failures where the fallback also drops remain an open question.
Benchmark and production are separate claims
The 99.20% figure comes from the Python evaluation harness, which measures whether the mechanism is correct. The Metriqual data plane is a separate Rust implementation with a zero-copy forwarding model and an in-memory conversation store. Its performance is an engineering claim, not one this paper evaluates, and the two are not reported together.
The harness is open. Run it against your own gateway.
The conversation set, the deterministic fault injector, both proxies and the judging pipeline ship together at tag v1.0-phase2-final. Point it at any OpenAI-compatible endpoint and it will tell you what your failover keeps.