Benchmarks
What moved our EnterpriseRAG-Bench score, and what didn't
How we reached our EnterpriseRAG-Bench score. The method, the three findings that mattered, the ideas that failed their own test, and what the benchmark cannot measure.
Louvain scores 91.32% on EnterpriseRAG-Bench. The well-known systems the benchmark's authors measured top out at 72.4.1
This post is how we got there, including the parts that did not work, because the method is more useful to you than the number.
EnterpriseRAG-Bench is published by Onyx: 500 questions over about 512,000 documents of a synthetic company, across chat, mail, tickets, wikis, drives, code and meeting transcripts.2 Its distractor documents are deliberately close to the right ones. They share names, customers and topics, and differ in what they say.
Where we started#
Our first run this month scored 72.26, about 12 points below our own October run of 84.05 on the same questions. Nothing about the benchmark had changed. We had:
- The answering tool had moved from one pass to a multi-turn loop with its own time limit and answer cap.
- Thirty-one settings had drifted between the two runs.
- A local environment file was overriding the benchmark's settings, so three runs we had labelled "October settings" read 10 documents per question, not 15.
We fixed each one and measured every change against the same questions.
The method#
Four habits did most of the work.
Attribute every loss to a stage. For a wrong answer, we did not ask "was retrieval bad?" We replayed the model's real searches and recorded where the right document sat at each stage: the keyword index, the vector index, their fusion, the reranker, what the tool showed the model, and what the model opened. A loss belongs to the first stage that dropped the right document.
Set the kill criterion before the test. Every idea got a written bar before it ran: "if fewer than four of these six questions move, the idea is not supported". It is the only defence we know against wanting a result.
Single levers detect harm; stacks show gains. Changing one thing at a time reliably told us when a change made answers worse. It rarely showed a gain on its own. Showing the model 24 results instead of 10 was flat alone, at twice the cost. Inside a combination, with the extra results shown compactly, it was part of every good configuration.
Ask a second model for a second opinion, with different evidence. A second model reading the same evidence mostly agrees with the first, as we learned the hard way below. A second model working a different way, with no shared assumptions, found our worst bug.
Finding 1: what the tool shows the model#
The replay of 88 real searches found 11 where the right document never reached the model. In five, it had been found and ranked 12th to 22nd while the tool showed 10. In two, the model had used a list view sorted newest first. Only four were true misses. The reranker never lost the best case it was given.
The other side of this is the most useful number we found. When the right document was the first one the model opened, the answer was right 118 times in 120. Nearly everything left is ranking at the very top of the page.
Finding 2: dates that weren't dates#
The benchmark's documents carry no dates; only mail and meeting transcripts have real ones. Our loader stamped a placeholder, 1 January 2026, on 374,507 of the 511,962 documents, 73% of them, and turned a year in a title into 30 June of that year. The marking that said "date unknown" was never stored. So our search and read tools showed the placeholder as a real date, and our guidance told the model to prefer the newer source when sources disagree.
On "which value is current" questions, the model was told the document holding the old value was the newer one. Ten of 21 date-filtered searches excluded the right document. The recency signal was flat for 286,000 chat messages.
This finding taught us the limit of second opinions. A self-check, a different model and a structured comparison of the candidates all agreed with the wrong answer, because all of them were shown the same wrong dates. Re-running one of those readers with honest dates flipped a question that had been wrong in every run.
Nobody's tooling found it. Someone read one sentence in a status report, "the right document's recorded date is 1 January", and asked whether that was a bug.
Finding 3: the stemmer#
Our keyword index reduced every word to its stem. "Shipping", in 8,922 documents, merged with ship, shipped and shipment across about 59,000. "Vectorization", in 392 documents, folded into "vector", in more than 13,700. A question about a shipping customer's rollout checklist returned ten generic rollout runbooks.
The second opinion that found it was another model replaying our own queries against a plain whole-document full-text index, with no model in the ranking at all. It found the right document for three questions we had scored zero on twice. An unstemmed field beside the stemmed one put the right document second for the checklist question. The cost was a 6.5-minute index rebuild and an index that grew from 814 MB to 1.1 GB.
What worked, in order of size#
- Showing 24 ranked results with a compact tail, instead of 10. Five of the eleven "never seen" cases were ranked 12th to 22nd.
- Answer behaviours learned from the leaders' published answers. Open with the answer and its identifiers. Check a document is about the same subject before answering from it. Commit to the governing source and say why. Cite only what the answer rests on. Worth 3 to 4 points on the same build. We used no system's answer text.
- Telling the model its last turn is its last. Five to seven questions per run had ended on "I'll check a few more documents".
- Keyword fixes. "SOC 2" now finds "SOC2", and "token-hours" is no longer two common words.
- One query embedding per search instead of 2.3. Per question, 8.7 embeddings became 2.6, which also let three runs fit on one machine at once.
- A filter that says so. A search filtered to an app nothing readable comes from now says that, instead of returning an empty list. 43 of 749 searches had died that way.
| Stage | Score |
|---|---|
| October run | 84.05 |
| Start of the day | 72.26 |
| First steady mode | 80.56 |
| Single fixes | 82 to 83.6 |
| Stack with answer behaviours | 85.05 |
| An earlier configuration | 88.59 |
| Latest configuration, 13 November | 91.32% |
What did not work#
Each test had its bar set before it ran. The failures are what a team trying the same thing most needs to know.
| Idea | What happened |
|---|---|
| Fixed headers on each chunk | No more right documents in the reranker's top 30 |
| Query expansion from the top results | Helped at most one search in twelve |
| Grammar-chosen required words | None of four |
| Learned sparse retrieval and late-interaction rerank | One more on question text, none on the model's real queries |
| Boosting exact names and entities | The distractors share the names by design |
| A one-line model summary on each hit | No gain |
| A model-written description per document | Helped far-out documents, flattened near-identical ones. At $37 to $148 for the corpus, cost was not the blocker |
| Majority vote across runs | The common answer was the wrong one |
| A same-model self-check | Re-endorsed its first reading, and broke four right answers |
| A reader that compares rival candidates | Agreed with the wrong answer while the dates were wrong |
| Five commercial embedders against our tuned open one | Some rank better alone. After fusion and reranking, ours put the most right documents on the page: 16 of 19 hard cases in the top ten, against 11 to 15 |
| Protecting the top fused results from the reranker | The reranker earns its place: 77 right documents in the top ten with it, 71 without |
| Scoping searches by source type | Two questions affected; parked |
| A different agent model, GPT-6.1 Sol | At low and medium reasoning, 7 and 5 points below Claude Sonnet on the same mode, at a quarter of the cost. It reached two questions no Sonnet run had |
What it changed in the product#
Every fix went into the product, not into a benchmark branch, because each one is something a customer would have met:
- Search shows more ranked results, with a compact tail, so the right document is not just out of sight.
- The keyword index matches "SOC 2" to "SOC2" and keeps an unstemmed field beside the stemmed one.
- An unknown date is never shown as a date. A date filter lists the strong matches it dropped. Newest-first puts undated documents last.
- A filter that can match nothing readable says so.
- One embedding per search.
- A stall in the titled-document search lane, about eleven seconds, is gone.
- A slow plan for a filtered listing, about 2,000 times slower in a first version of a fix, was caught by measuring rows and buffers before it shipped.
What the benchmark cannot measure#
Every question in EnterpriseRAG-Bench is asked by someone who may read every document. A company is not like that.
In Louvain, every one of these answers was produced the way every answer is: under the asking person's access, resolved before anything is searched. On this benchmark that person can read everything, so the fence never narrows anything and cannot be what moved the score. In a company, it is the property that matters most. Two people asking the same question should each get a true answer built only from what they may read, and a source one of them cannot open should change nothing they see.
We test that separately, with two readers on one corpus.3 A retrieval score tells you a system can find things. It does not tell you whether it should have shown them to you.
What we are doing next#
- Tune an open reranker and a larger open embedder on the corpus's own near-miss documents. Untuned, open rerankers already sit within one or two documents of the paid one.
- Make "which value is current" a state the product keeps, not a judgement the model makes each time.
- Keep the benchmark as a test of the product, run against every change that touches search.
Notes
-
The EnterpriseRAG-Bench leaderboard, on Hugging Face, read 11 November 2026. ↩
-
The benchmark's method is in its methodology file. ↩
-
The fence benchmark: same question, two permission sets, two different true answers. ↩