Practical guide · · By Raz Dagan
Your LLM-to-code evaluation needs an output comparator, not just a pass rate
Define which output differences matter before scoring a code replacement: JSON structure, missing fields, numeric values and unsupported inputs.
Write the acceptance contract before you count matches
Two systems can produce different text and still return the same useful result. They can also return similar-looking JSON with a business-critical difference. Before evaluating a replacement, specify the output contract and the decisions its consumer makes. A pass rate without that definition is difficult to interpret.
Consider a hypothetical call that extracts an order identifier and a quantity into a JSON object. Whitespace around the serialized object may not matter. A missing quantity does. An identifier with leading zeros may need to remain a string. These are application choices to document, not universal normalization rules.
Parse structure without erasing meaning
If the contract is JSON, parse the output and validate the expected shape before comparing values. Object member order can be irrelevant to an application while array order remains meaningful. Do not sort every array simply to increase agreement: a ranked list or sequence of steps can change meaning when reordered.
Define treatment of unknown keys, extra prose, duplicate keys and unexpected types. Some parsers accept inputs that a stricter contract rejects. A comparator should not silently repair malformed output unless that repair is an explicit part of the production path. Keep the raw result available for investigation under the existing retention rules.
Make missing, null and zero separate test cases
An absent field, a null value and a numeric zero can represent different states. If quantity is missing, the application might request clarification. If it is zero, the application might reject an empty order. A comparator that fills every missing field with zero can turn a disagreement into a false success.
Use synthetic fixtures that isolate each case: an omitted quantity, an explicit null, the number 0 and the string "0". Write the intended result for each before running the candidate. Review the consumer as well as the producer; a type conversion downstream may conceal a difference during a simple visual review.
Apply tolerances only where the contract permits them
Exact equality is appropriate for an identifier or fixed label. A calculated measurement may need a documented numeric tolerance with a unit. Do not introduce a blanket tolerance across every number. An amount of money, a count and a measured physical value have different requirements.
For text, avoid stripping case, punctuation or Unicode distinctions globally. Decide whether the task promises an exact code, human-readable prose or a normalized search key. If a field requires semantic judgment, label that part of the evaluation separately rather than pretending that a deterministic string comparator resolves it.
Report invalid outputs and disagreements alongside the score
Keep counts for valid matches, valid disagreements, invalid outputs and explicit unsupported cases. Record the evaluation-set version and comparator version with the result. A change to either can move the score even when the candidate code is unchanged. Re-run the fixed held-out set after a comparator correction.
Inspect disagreement examples before treating the previous model response as the correct label. Agreement measures compatibility with that reference; correctness requires an independently reviewed contract or labels. Do not select a normalization rule merely because it makes the candidate look better.
Use the comparator at the integration boundary
JavaScript and Python JSON behavior is documented separately. If you port a candidate, test the actual runtime and serializer rather than assuming identical behavior. MDN documents JSON.stringify and the Python standard library documents its JSON encoder and decoder; consult them for the values your application accepts. Sources: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify and https://docs.python.org/3/library/json.html.
ModelGate code replacements run in the customer application; Python ports are not executed by ModelGate. Use a held-out evaluation before integration, and carry the same acceptance contract into application tests. Review the product demo and documentation to assess the workflow without turning an illustrative comparator into a claim of measured savings.
Explore the workflow
Read the code-replacement requirements, GitHub workflow and gateway comparison before deciding.