Evaluating Search Relevance the Right Way

Moving Beyond Gut Feelings

When we build search features, it is easy to fall into a trap. You run a query, look at the results, and decide if they look good enough. This is how most of us start, but it is a terrible way to measure success in the long run. If you cannot measure search relevance objectively, you are flying blind.

Why Basic Testing Fails

Most teams rely on simple unit tests for search. They verify that a specific keyword returns a specific document. This is fine for catching regressions, but it tells you nothing about the quality of the results. Search is probabilistic, not deterministic. You need to evaluate the rank order of your results, not just the presence of a hit.

Using Mean Reciprocal Rank

One of the most useful metrics is Mean Reciprocal Rank (MRR). It focuses on where the first relevant result appears. If your user finds the best match at rank one, you get a score of 1. If it appears at rank two, you get 0.5. If it is at rank five, you get 0.2.

Here is a simple way to calculate this in JavaScript:

1
function calculateMRR(results, targetId) {
2
  const rank = results.findIndex(result => result.id === targetId) + 1;
3
  return rank > 0 ? 1 / rank : 0;
4
}
5

6
const searchResults = [{id: 'a'}, {id: 'b'}, {id: 'c'}];
7
console.log(calculateMRR(searchResults, 'c')); // 0.333

The Importance of Human Judgement

Metrics like MRR or Normalized Discounted Cumulative Gain (NDCG) require a ground truth. You need a set of test queries paired with the documents that users actually want to find. If you do not have logs, create a small set of golden queries. These are queries where you manually label the results as relevant or irrelevant.

Automating Evaluation

Once you have your golden queries, you need to run them against every change you make to your search index or ranking algorithm. If you change your tokenizer or tweak your term boosting, you should run your suite to ensure you have not degraded your results.

Common Pitfalls to Avoid

Do not optimize for the wrong thing. Focusing too much on exact keyword matching often ignores intent. If your user searches for ‘laptop,’ they probably want a product page, not a blog post about ‘how to fix a laptop.’ Your search evaluation must mirror the user intent. If you ignore the context of your data, your search relevance will suffer regardless of how fancy your algorithm is.

Keep your evaluation pipeline simple. Start with MRR, build a small set of golden queries, and make sure those tests run in your CI pipeline. If you find yourself wondering if your search is good enough, stop looking at the results and start looking at your metrics.