The usual way a RAG system gets evaluated: someone asks it a few questions, reads the answers, and says "that feels better." Then a chunking change improves three answers and quietly breaks forty.
Two numbers end the guesswork, and they measure different halves of the system.
Recall@k asks whether the right passage made it into the context at all. That is retrieval. Faithfulness asks whether the answer is actually supported by what we sent. That is generation. An answer can be fluent, confident, and unsupported by every passage it was handed.
Which is how one change reads both ways at once. Recall@5 drops from 0.72 to 0.68 while faithfulness rises from 0.78 to 0.80 — smoother prose over thinner evidence, and by eye it looks like progress.
What makes any of this possible is dull: 100+ real questions with the passage that answers each one, run after every change.
Improvement needs a baseline. Without one we aren't improving — we're just changing things and hoping.