Practical guide · · By Raz Dagan
Before replacing an LLM call, separate state, side effects and transformation
Map which parts of an LLM workflow can become a deterministic function and which still need application state, permissions or external actions.
Draw the boundary around one transformation
A recurring prompt may combine several jobs: read a customer record, interpret text, choose an action and write back to another system. Treating that whole prompt as one replaceable function hides responsibilities. Before generating code, list the inputs, outputs and side effects separately.
For a fictional support workflow, parsing a known status label is a transformation. Looking up the current subscription is a state read. Sending a refund request is a side effect. The fact that these steps appeared in one model conversation does not mean one generated function should own all three.
Make changing state an explicit input
If a decision depends on account status, inventory or policy version, decide where that value comes from and when it is read. A function cannot infer the current state from an old example. Pass a validated snapshot or keep the state-dependent step in the application.
Write down freshness and missing-value behavior. For example, an absent policy version could require rejection rather than a default guess. A deterministic implementation can be reproducible while consistently applying stale information; determinism alone does not establish business correctness.
Keep permissions at the application boundary
A transformation that returns an action label should not acquire authority to perform that action. Check permissions in the application using the actual authenticated context. Do not turn text such as 'approved by admin' into authorization because it appeared in training examples.
Review output consumers too. A downstream service may treat a category as an instruction. Specify allowed outputs and validate them before execution. Tests should include unrecognized labels and malformed input, with the expected rejection or fallback path made explicit.
Prevent duplicate effects during comparison
Shadow evaluation should compare outputs without repeating customer actions. If both old and new paths send messages or update records, the experiment changes the system it is measuring. Isolate a pure decision where possible and route the chosen result through one controlled action path.
Use synthetic or appropriately sanitized examples during development. Do not copy private records to a new service merely to increase evaluation coverage. Preserve the original consent, access and retention boundaries and record which data is actually required.
Evaluate the bounded candidate
Freeze the input contract and evaluate the candidate on held-out cases. Include changes in state, unsupported versions, missing fields and failures in the surrounding dependencies. Agreement with old model outputs is one signal, not a substitute for reviewing the business rule.
ModelGate's code replacement runs in your application. Python ports are not executed by ModelGate and require separate validation. Use the documentation and demo to assess the workflow, then review the integration boundary, permissions and side effects before production release.
Explore the workflow
Read the code-replacement requirements, GitHub workflow and gateway comparison before deciding.