Keyword search vs embeddings + reranking, measured on 648 real questions.

What this means if you are not an IR person

A RAG system or an AI agent can only answer from what its search step hands it. If the right document is not in the top 10, no prompt and no model upgrade will produce the right answer. So retrieval quality is the ceiling on the whole system, and it has to be measured, not assumed.

Keyword search (BM25 — what most databases and “add search” plugins give you) finds the right answer in the top 10 for about 30% of these questions. Swapping in a small embedding model lifts that to 46% — the single biggest win, and it costs one millisecond. Adding a cross-encoder reranker on top takes it to 49% and, more visibly, pushes the right answer nearer the top: nDCG@10 goes 0.233 → 0.429, and the first correct passage moves up the list (MRR@10 0.287 → 0.516). 158 of the 648 questions go from “no correct answer shown” to “correct answer shown”; 23 go the other way.

That is the difference between an assistant that is right most of the time and one that confidently answers from the wrong page.

Results

Latency is end-to-end per query on an M2 Max laptop (Metal, no GPU server, no caching), so treat it as relative, not absolute. The reranker is the expensive stage: 2331 ms here for a 568M-parameter model scoring 50 candidates. On a server GPU the same pass is tens of milliseconds, and cutting the candidate list from 50 to 20 roughly halves it again — it is the lever you tune per product, not a fixed cost.

What did not work

Hybrid retrieval hurt. Fusing the keyword and embedding lists (reciprocal rank fusion, the standard recipe) scored below embeddings alone — 0.420 vs 0.464 recall@10. On this corpus BM25 is weak enough that blending it in drags the good ranker down. A small reranker did not rescue it: our first run reranked the hybrid list with the older BAAI/bge-reranker-base and reached only 0.431 recall@10 / 0.350 nDCG@10 — still below plain embeddings. The numbers in the table come from the second run: rerank the best retriever’s candidates with a current-generation model. Both runs are in the repository. This is the point of measuring: the default recipe is not always right, and you cannot tell without the number.

How to read the columns. recall@10: of the passages labelled relevant for a question, what share appeared in the top 10. nDCG@10: the same, but rewarding results that appear at rank 1–3 over rank 8–10. MRR@10: on average, how high the first relevant passage sits. recall@50: what the reranker had to work with — it can only promote what the retrievers found.

Method

• Dataset: BEIR FiQA-2018 — 57,638 passages from financial Q&A forums, 648 test questions, 1,706 human relevance judgments. Used exactly as published; no cleaning, no filtering, no query rewriting.
• Keyword search: BM25 via bm25s, English stopwords, default k1/b. Top 100.
• Dense: BAAI/bge-small-en-v1.5 (33M parameters), cosine similarity over normalized embeddings, BEIR query instruction prefix. Top 100.
• Hybrid: reciprocal rank fusion (k=60) of the two lists above. Top 100.
• Embeddings + rerank: the dense top 50 re-scored by the cross-encoder BAAI/bge-reranker-v2-m3 (568M parameters), then cut to 10.
• Metrics: computed with ranx against the official test qrels. Deterministic; no seeds, no cherry-picking of runs — the first run is reported above under “what did not work”.
• Hardware: Apple M2 Max, PyTorch on Metal. Embedding the corpus took 154.8 s; building the BM25 index 3.8 s.

Caveats, stated plainly. A public benchmark is not your data; the relative gap between systems transfers better than the absolute numbers. Relevance labels in FiQA are incomplete (some genuinely relevant passages are unlabelled), which depresses every system’s score equally. Latency was measured on a laptop with other processes running.

Reproduce it

The harness is ~120 lines of Python and lives in the same repository as this website. Two commands, no API keys; about 30 minutes on an M2 Max, most of it the reranker (the model downloads are ~2.4 GB the first time):

git clone https://github.com/RerankLab/reranklab.com && cd reranklab.com/evals/fiqa
uv run run.py        # downloads FiQA (~20 MB) + two open models, writes results.json
uv run render.py     # turns results.json into this page

If your numbers differ from the ones above by more than rounding, tell us — we will either fix the harness or update the page.