Use prompt caching for repeated context and verify the cache hit
Keep reusable material stable, follow the selected model’s caching controls and check usage instead of assuming every repeat is discounted.
In brief
Prompt caching can reuse eligible repeated input context. It is different from saving a completed answer: the model still generates a new response, and eligibility depends on the provider and configuration.
A stable reference document reused across many tasks is a better candidate than unrelated short requests.
Structure repetition without adding filler
Put useful stable material in the reusable portion and changing questions afterward when the API supports that arrangement. Do not pad requests with meaningless text to cross a threshold. Changing the model or tool definitions can change matching even when the visible document appears identical.
Follow the current provider mechanism
OpenAI’s supported paths and models have specific caching and diagnostic controls; Anthropic and Gemini use their own mechanisms. Read the current documentation for the actual request you send. Do not copy one provider’s cache flag into another SDK and assume it works.
Measure misses and retention
Record eligible and cached input separately and examine repeated-request behaviour. A cache expires or is unavailable under conditions in the provider’s policy. Account for uncached requests and any explicit cache storage/write costs, without presenting a theoretical saving as a measured one.
Before you act
Model, tool, prefix, length and retention requirements vary. A cache miss can cost and behave like ordinary input; do not budget every token at the cached rate.
What to do next
- Identify genuinely repeated instructions or reference material.
- Keep the eligible reusable prefix and relevant settings consistent, using documented cache controls.
- Inspect cached-input usage and compare total cost and latency for real repeated requests.
