Practical guide · · By Raz Dagan
Exact caching or code replacement? Choose by what repeats in your LLM workload
Separate repeated requests from repeated tasks, estimate cache eligibility and replacement coverage, and choose an evaluation plan before changing production.
Start with the unit of repetition
Two workflows can both look repetitive while needing different optimizations. In one, the application sends the same request many times and can reuse an earlier answer. In the other, every request contains a different order or document, but the operation performed on it follows the same explicit rule. Exact caching addresses the first situation. A code replacement may address the second.
Write down whether it is the complete request or the underlying transformation that repeats. Do not use a shared prompt template as evidence of an exact cache hit. A template saying 'convert this status to a label' can be identical while the status, identifiers and required answer change on every call.
An exact cache reuses an answer within an identity and lifetime
For an exact cache, define what makes two requests equivalent for reuse. The answer can depend on the messages, provider, model and response-shaping settings, as well as the application or tenant context. Missing a relevant input can make a cache appear effective while returning an answer to a different question.
ModelGate's current exact-cache identity includes the project, provider, model, temperature, output-token limit, tools, response format and exact messages. Its lookup also checks expiry and project ownership. The current cache implementation defaults to a 24-hour lifetime. These are current product implementation details, not a recommendation that every application should reuse data for 24 hours.
A fixed transformation can repeat across different inputs
Consider a fictional order-status service. One request contains status 'shipped' for order A; another contains status 'pending' for order B. Reusing the complete answer from A for B would be wrong. If the only required operation is a documented mapping from the supplied status to a UI label, a function can apply that mapping separately to each input.
Before replacing the model, inspect what else the call does. If it resolves contradictory notes, reads changing external data or interprets a customer's free-form message, a lookup table may omit necessary work. Define missing-value and unsupported-input behavior instead of choosing a familiar label whenever a new value appears.
Measure cache opportunity without weakening request identity
Use a representative window of requests to count eligible repeats within the proposed lifetime. Separate identical requests from nearly identical ones. A request that changes a customer identifier or timestamp may need a different answer; removing those values just to increase a hit rate changes the correctness question.
ModelGate's cache eligibility helper currently requires temperature zero with no tools or response format. Eligibility does not prove factual correctness or freshness: it only describes which requests that optimization considers. Check the documented limits and your application semantics before enabling reuse. A service that reads changing inventory still needs an appropriate freshness policy.
Measure replacement coverage on examples the implementation did not use
For a code candidate, keep implementation examples separate from evaluation examples. Evaluate different inputs that exercise the same task, including malformed values and unsupported cases. Group near duplicates before splitting so repeated examples do not inflate agreement. Agreement with a recorded model response is not automatically independent ground truth.
Report both quality on handled inputs and coverage across the whole workload. A function that handles one narrow status format perfectly may still send most requests to a fallback. ModelGate's documented replacement process uses held-out examples; review its thresholds and disagreements. The supplied Python port is not executed by ModelGate and requires your own tests.
Compare costs using the path each request actually takes
For caching, account for eligible hits, misses, expiry, lookup overhead and storage. For a replacement, account for handled inputs, fallback model calls, execution, evaluation and maintenance. Keep the denominators visible: a hit rate on eligible requests and a coverage rate across all requests are different measurements.
As an illustrative calculation, a fictional batch of 1,000 requests might contain 100 reusable hits, while a separate function could handle 400 changing-input transformations. These invented counts explain the comparison only; they are not measured ModelGate results or additive savings. Some requests may belong to both groups, so evaluate their actual execution order before counting avoided calls.
Choose a small rollout with a reversible decision
Choose caching when safe answer reuse is supported by complete identity and freshness requirements. Investigate a replacement when a changing-input task has an explicit contract that tested code can satisfy. Keep the model when the missing work is language interpretation rather than a rule you can state and validate.
A replacement runs in the customer's application; it is not arbitrary code execution inside ModelGate. Keep the original path available while reviewing behavior and costs. Use the documented code-replacement requirements and gateway workflow linked below to choose one bounded task, rather than enabling several optimizations and guessing which change affected the result.
Explore the workflow
Read the code-replacement requirements, GitHub workflow and gateway comparison before deciding.