This is a small story about a small change that paid back faster than we expected. It's not a tutorial. The lesson is implicit; if you take it as one, you'll have to reverse-engineer it from what happened, the same way we did.
The morning it broke the wrong way
We have a cron that runs at 06:30 every weekday. It's the kind of scheduled run an operator eventually accumulates without quite meaning to: a daily check, a summary, a few outbound calls, a file written to a known location, a one-line ping. The kind of thing that, if it stopped running, someone would notice by lunchtime but no one would notice by 06:31.
For most of the summer, the first call of the morning landed in about 4 seconds. We knew this because we logged it, not because we watched it. 4 seconds is not a problem. 4 seconds, twice a day, against a model whose per-token cost is whatever it is, is not a budget item. We had stopped thinking about it.
Then one morning it landed in 1.1 seconds. The next morning, 1.1 again. The next, 1.1. By the third day we looked up.
What changed
Nothing we had done on the system side. Same model, same provider, same tool surface, same agent loop, same prompt file, same cron schedule. The thing that had changed was a small restructure of the prompt itself: a long static block — instructions, persona, the section we always paste in unchanged — had moved from somewhere in the middle of the prompt to the very top.
That was it. We had been editing a different section of the prompt earlier in the week and moved the static block upward as part of a tidying pass. We didn't expect anything to come of it. The block was the same content; only its position changed.
What we learned, after the fact, was that the provider's prompt cache keys on a prefix match from the start of the prompt. As long as the static block was in the middle of the prompt, every morning's run had to send the variable part first, which meant the cache hit rate was effectively zero. Move the static block to the top, and the morning run starts inside the cache window. The 4 seconds dropped to 1.1 because most of the tokens the model had to read were already in cache; the actual generation was the smaller, more variable part of the work.
The 14% cost drop is the same story from a different angle. Fewer tokens read, fewer tokens billed for input. The model and the call pattern didn't change; the cache hit rate did.
What we learned, slowly
The lesson isn't "move static blocks to the top of your prompts." That sentence is correct, but it's a tactic, and a tactic without a model is the kind of advice that ages badly.
The model is this: prompt cache hit rates are non-obvious. You can have a prompt that looks cache-friendly and isn't, because the variable part sits at the top. You can have a prompt that looks cache-hostile and is, because some prefix you've been pasting in unchanged for months is doing more work than you thought. The cache behavior is a property of where in the prompt the variability lives, not of how big the prompt is.
The corollary is that the cost of restructuring a prompt to be cache-friendly is paid back in days. The restructure took us about ten minutes. The savings, at our current call volume, are a steady 14% on the line item in question. By the end of the first week we were ahead. By the end of the second we had forgotten how the old prompt was laid out, and that was fine, because the new layout is now the only layout that exists in our heads.
The third thing we learned is the most uncomfortable. We had been paying the 4-second tax for months without noticing. The cost was small enough to ignore, the latency was small enough to ignore, and the failure mode of "the cache isn't hitting" is the kind of thing that doesn't show up on a dashboard. If we hadn't moved the static block by accident, we would probably still be paying it.
What didn't change
To be honest about what we learned, we have to be honest about what we did not change.
The model is the same. The provider is the same. The tools the agent uses are the same. The agent loop — the intake, context, inference, tool execution, persistence, termination sequence — is unchanged. The cron schedule is unchanged. The success criteria are unchanged. The one variable that moved was the position of a static block inside the prompt. Everything else is a control.
This matters because the temptation, on a day when latency drops, is to attribute the drop to whatever was last touched. Most days, whatever was last touched is the wrong explanation. In this case, the only thing last touched was the prompt layout, and the behavior change matches the prompt layout change cleanly enough that we can be confident about the cause. On a different day, we would not have been.
What we check now, weekly
We added three things to the weekly review. None of them are elaborate.
1. Cache hit rate on the hot paths. The morning cron and the two other prompts that account for most of the daily input volume. We log the cache hit indicator (the provider returns one) and we look at it on Monday. If a hit rate drops below a threshold we set empirically — currently 80% for these prompts — we look at the prompt layout before we look at anything else. 2. Position of the static blocks. When we edit a prompt, we ask whether the prefix is still cache-friendly. Not always; some prompts are too short or too variable for the question to matter. For the long ones, the question is now a default part of the review. 3. The shape of the latency distribution, not just the average. Averages hide the morning spike. We had been looking at averages; the cache miss showed up as a long tail that didn't move the average much. The first-call latency is now logged separately, so a regression would be visible on a single morning rather than over a quarter.
None of these are sophisticated. The sophistication was already in the provider; we just weren't looking at it.
What this is not
This isn't a prompt-engineering essay. The prompt-engineering piece covers the broader pattern; this field note is one small observation from one morning's cron. The general rule is bigger than this story.
It isn't a recommendation to switch providers. Some providers cache more aggressively than others, and the right choice depends on a dozen other things. The change we made is the change we could make without changing providers.
It isn't a guarantee. If we restructured the prompt again, we might move the static block somewhere that breaks the cache, and we'd find out on a Monday morning when the average latency crept back up. The check exists because the mistake is easy to make.
The honest summary is the one that doesn't have a moral at the end. We moved a static block to the top of a prompt. The morning got faster. The cost went down. We now look at the cache hit rate once a week. That's the whole story.
— Triadive Editorial