← All articles

Practical guide · · By Raz Dagan

What to log before replacing an LLM call with deterministic code

Define a minimal, privacy-aware observability contract for eligibility, disagreements, fallbacks and runtime failures before rollout.

Start with decisions, not a request dump

A replacement needs evidence about what path ran and why. Log the eligibility decision, replacement version, fallback reason and final outcome instead of storing every prompt and response by default. That gives an operator a usable trail without turning observability into a second copy of sensitive inputs.

Write down the questions the log must answer: was the request eligible, did the generated code return a valid result, did the comparator disagree, and did the application fall back to the model? Fields that do not answer an operational question should need a separate justification.

Give every replacement a stable version

Record a replacement identifier and version with each execution. Without it, a change in generated code can look like random production drift and cannot be tied back to the evaluation that approved it.

Keep the evaluation artifact, code review and deployment version linked. ModelGate generates a replacement that runs inside the customer's application; it does not execute Python ports on the customer's behalf, so application-side telemetry must identify what actually ran.

Separate eligibility from success

A request can be ineligible and correctly routed to the LLM. That is not a replacement failure. Track eligible volume, replacement attempts, completed replacements and fallbacks as separate counts.

Coverage should be calculated from a defined population. If unsupported inputs silently disappear from the denominator, a high success rate can hide that most production traffic still uses the model.

Capture disagreements without exposing raw content

During shadow or canary evaluation, record comparator categories such as exact match, tolerated numeric difference, missing field or semantic review required. Prefer hashes, bounded labels or redacted samples when raw values are not necessary.

If a disagreement needs investigation, use an access-controlled workflow with retention limits. A general analytics stream should not become permission to retain prompts, customer data or model outputs indefinitely.

Make fallback reasons actionable

Use a small controlled vocabulary: timeout, schema failure, unsupported input, comparator rejection, dependency error or manual rollback. Free-text reasons are difficult to aggregate and can accidentally collect sensitive details.

Measure both replacement latency and fallback latency. A fast failure followed by a full model call may increase end-to-end latency even if the generated code itself is inexpensive.

Test the telemetry on held-out and failure cases

Run held-out examples through the same instrumentation that production will use. Verify that unsupported inputs, deliberate exceptions and comparator mismatches produce the intended events without changing the application result.

Before rollout, prove that operators can answer the rollback question from the logs alone. If the signal cannot identify a bad version or route traffic back safely, the observability contract is not ready for a canary.

Explore the workflow

Read the code-replacement requirements, GitHub workflow and gateway comparison before deciding.