Practical guide · · By Raz Dagan
When to replace LLM calls with code: a practical evaluation guide
Identify rule-based LLM tasks, test a code replacement against held-out examples, and calculate savings after fallbacks and maintenance.
Start with a bounded task, not a promise to remove AI
A production workflow often mixes reasoning with routine transformations. A language model may help interpret a message, while a later call only rearranges fields or returns a value already present in the input. Evaluate that second step independently. Replacing a narrow transformation can be useful without changing the part that needs language understanding.
Write down the input contract, the expected output and the consequences of an incorrect answer. A JSON field mapping is a stronger starting point than open-ended support advice. A small set of output labels is a clue to investigate, not proof that keyword rules will work: a sentiment classifier can have three labels and still require understanding context.
Choose candidates from real requests
Group recorded requests by the task they perform, rather than by model name alone. Look for repeated extraction from a known format, arithmetic over explicit numbers, schema conversion and validation against an existing rule. Measure how often the task runs and what those calls cost before prioritizing it.
Avoid using a convenient sample containing only successful or short requests. Include empty values, malformed input, Unicode, conflicting fields and unusually long examples. If the task reads external state, include how fresh that state must be. A replacement that silently ignores a lookup can agree with yesterday's examples and fail tomorrow.
Separate implementation examples from evaluation examples
Keep a set of examples out of the implementation process. Use one portion to write or generate the function, and another to evaluate it after its behavior is fixed. Otherwise the score can reflect memorization of the examples used to build it. Keep related or nearly identical requests together when splitting, so a duplicate does not leak into both sets.
Define the acceptance rule before running the evaluation. Exact string equality may suit a fixed label. A structured response needs parsing and schema checks; object key order should not count as a failure, while a missing required field should. Review disagreements manually. Agreement with the previous model measures consistency with that model, not independent correctness.
A concrete example: mapping an order status
Suppose an internal service supplies a documented status field, and a model call translates pending, shipped or delivered into three fixed UI labels. A lookup table can perform that mapping. First verify that the model is not also resolving contradictory status history, translating free text or applying another business rule that the status field alone cannot express.
Specify the behavior for a missing or unknown status. Returning an explicit unsupported result lets the caller choose a fallback. Guessing a familiar label may make an evaluation look better while concealing a production error. Test each supported value, unknown values, unexpected types and changes to the upstream contract. This is an illustrative workflow, not a customer result or a measured saving.
Compare total cost and operational behavior
Calculate the provider spend avoided, then subtract the extra runtime, evaluation, monitoring, fallback calls and maintenance costs. A local function has no model-provider charge for that operation, but compute and engineering work still cost money. A fallback-heavy replacement may save much less than the headline number suggests.
Track latency distributions and errors alongside cost. Run the candidate in shadow mode where appropriate: compare its answer without changing the answer users receive. Follow with a limited rollout and a clear rollback path. Monitor the percentage of inputs the replacement actually handles, because a high match rate on a small subset can conceal low coverage.
Where ModelGate fits
ModelGate's documented code-replacement workflow uses recorded input/output pairs to identify candidates and verify replacements. Prompt storage and response capture are required. Its current documentation describes a minimum of 50 recorded calls and a verification threshold of at least 95% agreement on at least 30 held-out answers. Those thresholds describe the product workflow; they are not a universal quality standard for every task.
Review the generated code and its disagreements before merging a pull request. The replacement runs in your application rather than becoming arbitrary code execution inside the gateway. TypeScript recipes contain the verified logic; the supplied Python port is not executed by ModelGate and needs your own tests. Check the documentation for the current requirements and limits before starting.
Decide whether to ship, revise or keep the model
Ship only when the candidate meets the task's acceptance criteria and its total cost is worthwhile. Revise it when failures expose a missing but explicit rule. Keep the model when the task depends on broad language understanding, evolving context or unsupported inputs that would turn the function into an unreliable collection of exceptions.
Start with one task, record the baseline and keep the original path available during rollout. The useful outcome is a repeatable decision with evidence: which inputs the replacement handles, how it was tested, who owns it and how to reverse the change.
Explore the workflow
Read the code-replacement requirements, GitHub workflow and gateway comparison before deciding.