Your assistant answers what's our refund policy? — four thousand tokens of context, two seconds, a few cents. Then it answers it again. And again. Four hundred times that week, from different employees, against the identical retrieved documents. Full price every time, for an answer you already had.
Two caches sit at different depths.
Prompt caching happens at the model provider. Your long system prompt and retrieved documents get stored, so the same prefix next time is far cheaper and the first token arrives much faster. A fresh answer still gets generated.
Semantic caching you run yourself. Store question against answer, embed the new question, compare. Close enough to a past one and you return the old answer with no model call at all — the largest saving available, in both cost and latency.
Close enough is the catch, because it's a similarity threshold you chose. "Can I refund an order?" and "Can I cancel an order?" score 0.92 against each other and want opposite answers. Speed bought with staleness is the familiar trade, except here the stale thing might be subtly wrong rather than merely old.
Caching meaning is harder than caching data, because two things can look similar and mean opposite things.