Reliability · the third axis
Cost tells you what your bot spends. Reliability tells you whether you can trust it.
The gateway already sees every request and response. That makes it the one place to answer two questions no dashboard usually can: is my bot stable — does it answer the same task the same way, keep its format, and avoid contradicting itself — and what’s confusing it, and where. No app changes, and the evidence leads: never a lone percentage.
See it
The same task, asked four ways. Watch the score and the flags move.
The same labelling task, asked four ways. A reliable bot gives one answer regardless of phrasing.
Classify the sentiment. Reply as JSON {label}.Computed in your browser with the same shape of logic the gateway applies — a consistency score plus deterministic detectors, no model call. The score is an estimate; the flagged answers are the evidence.
Two jobs
Is my bot stable? A consistency score, computed continuously from your live traffic and tracked as a daily trend, with a regression alert when it drops. Under the score sit the deterministic detectors — format breaks, degenerate output, refusals — that flag individual failures as they happen.
What’s confusing it, and where? A confusion map that correlates failures against the things that tend to cause them — thread length, the specific task, the model — and ranks the triggers with their support counts. So instead of “reliability is 82%” you get “73% of failures happen on conversations past eight turns.”
What it detects
Two families. Deterministic detectors are cheap, high-confidence, and run inline on every response — they are the free hook. Judge detectors read a whole conversation with a model call, so they need response capture switched on and are part of the add-on. All of it is off or in detect until you decide otherwise.
| Detector | What it catches | State |
|---|---|---|
| Format break (deterministic) OFF → DETECT | When a response was requested as JSON or a schema and the answer doesn't parse. High-confidence, no model call. | Free · always on |
| Degenerate / looping (deterministic) OFF → DETECT | Empty responses, truncation, and runaway repetition loops — the classic 'the model broke' signatures. | Free · always on |
| Refusal (deterministic) OFF → DETECT | Canonical assistant-refusal phrasing. On its own a refusal may be correct; a shift in the refusal rate for a task is the signal. | Free · always on |
| Consistency score (deterministic) Always computed | Mean pairwise agreement across a task's answers — using the response hashes when text isn't stored, the text when it is. This is the passive score and its daily trend. | Free · always on |
| Self-contradiction (judge) OFF → DETECT → ENFORCE | Reads a whole conversation and flags when the bot says X and later asserts the opposite. Needs response capture and one model call. | Add-on |
| Instruction drift (judge) OFF → DETECT → ENFORCE | Flags when the bot quietly stops obeying its own system rules as the context grows. | Add-on |
| Grounding (judge) OFF → DETECT → ENFORCE | For retrieval bots — flags an answer that asserts facts the provided context doesn't support. | Add-on |
| Active Consistency Test Metered per run | Fires N paraphrased variants of one prompt through the same model and scores how much the answers disagree. A CI-callable regression gate. | Metered |
Evidence first, estimates second
The consistency score is an estimate, and we treat it like one. Every incident on your dashboard leads with the actual evidence — the divergent answers side by side, the conflicting turns highlighted — and the percentage is a label on top of it, never a substitute for it. Judge verdicts carry a confidence and the spans that justify them, not a bare number.
How it installs — nothing changes in your app
You already route through the gateway. Reliability adds two optional, honest touches:
- Conversations. To reason about a whole thread (contradiction, drift) the gateway needs to know which turns belong together. Pass a
x-modelgate-conversation-idheader and it’s exact; omit it and the gateway stitches turns heuristically by overlap and time. Either works. - Response capture. The judge detectors and the evidence drill-down need the answer text. It’s off by default, opt-in per project, and retention-capped — a purge job clears stored responses past your window automatically. The deterministic score needs none of it.
The regression gate
An active Consistency Test fires several paraphrased variants of one prompt through the same model and measures how much the answers disagree — 100 means every phrasing gave the same answer. It returns a single varianceScore, so your CI can fail a deploy when a change makes a critical prompt less stable:
curl -s -X POST "$MODELGATE_API/v1/dashboard/reliability/test" \
-H "x-api-key: $MODELGATE_KEY" -H "content-type: application/json" \
-d '{"promptTemplate":"<your critical prompt>","model":"gpt-4o-mini","provider":"OPENAI","variantCount":6}' \
| python3 -c 'import sys,json; v=json.load(sys.stdin)["varianceScore"]; sys.exit(0 if v>=80 else 1)'Active tests spend the customer’s provider tokens through their own stored key, so they’re metered. See pricing for the add-on and per-run figures, and the docs for the full API.
What we don’t claim
- The detectors are heuristics and a judge model — not proofs. They catch the common failure families, not every one.
- The consistency score is an estimate derived from agreement; it is not a correctness guarantee.
- Judge detection depends on a model call and inherits that model’s limits — read the evidence, not just the verdict.
- No benchmark scores, no invented reliability percentages for models we don’t run.