Two ways to find a case
AI research tools look up authority in one of two ways: by matching the words you typed, or by matching what they mean. The two behave differently on legal text, and the difference has been measured. This page explains both, shows what the published studies found, and ends with how Passim AI searches.
Word matching, done carefully
Lexical search is keyword search with a scoring formula behind it. It’s older than the web and it still runs most of the search you use.
A lexical index records, for every word in the library, which documents contain it and how often. A query is scored by the words it shares with each document. The formula most engines use, BM25, adds two refinements that matter in law. A word that appears in few documents counts for more than a word that appears everywhere, so recklessness outweighs violence. And a passage that uses a word several times in a short span scores higher than a long opinion that mentions it once in a factual recital.
That is the whole mechanism. There’s no model of what the words mean, which is both the limitation and the reason it’s trustworthy: a result matched because it contains the words, and the engine can show you which ones.
What it does well
- Exact strings match exactly. Citations, docket numbers, statutory subsections, party names, and terms of art are found by their characters. Nothing has to understand them.
- You can see why. The matched words can be highlighted in the result. In a profession that verifies everything, that’s a feature.
- New decisions are searchable the moment they’re indexed. No model has to process them first, and the index doesn’t go stale when a model changes.
- Long documents are searched by passage. A forty-page opinion is scored paragraph by paragraph, so the one paragraph that matters is enough to find it.
Where it fails
- Different words, same idea. A question about being “deported for drunk driving” shares no words with 8 U.S.C. § 1227(a)(2), which answers it in terms of convictions, moral turpitude, and aggravated felonies.
- Stems and spellings. Parole and parol share a stem. Protección loses its accent in some tokenizers. Scanned decisions carry OCR errors that break words.
- No sense of the question. A fact pattern that calls for a rule the asker can’t name yet gives the engine nothing to match.
RAG, and what people mean by it
Retrieval-augmented generation is a design, not a search method. The search step inside it can be lexical or it can use embeddings, and the vocabulary around it has gotten loose.
In a RAG system the language model looks things up before it answers, and the answer is built from what came back, with citations to it. That is a sound design for legal work: the model is reading current authority instead of reciting what it remembers from training. The part that varies is the retriever. It can be a lexical index like the one above, or an embedding index, or both. When a legal AI product says it uses RAG or semantic search, it almost always means the embedding kind, and that is what this page compares against word matching.
How embeddings match
An embedding model converts a passage into a long list of numbers that places it in a space where passages with similar meaning sit near each other. Your question is converted the same way, and the search returns the nearest passages. Nothing is matched word for word, which is the point: “drunk driving” and “operating a vehicle while intoxicated” land close together, and a question in plain English can find a statute written in none of the same words.
The model learned what “similar” means from the text it was trained on. For general-purpose embedding models, that text is the open web. The studies below turn on that fact.
What it costs
- The model decides what counts as similar. To a web-trained model, asylum and withholding sit close together, and so do inadmissible and deportable. Those are the distinctions legal research turns on.
- One vector per chunk. Text is cut into chunks and each gets a single vector. Compress a long opinion into a few hundred numbers and the paragraph that matters is averaged away.
- Strings blur. A citation or a section number isn’t a meaning. Models place 27 I&N Dec. 316 near other citations, not near the one decision it names.
- Nothing to show. A nearest-neighbor score can’t be explained by pointing at words in the text.
- Rebuilds. Changing the embedding model means re-encoding the whole library.
- Hallucination remains. Retrieval narrows what the model sees. It doesn’t make the model read correctly. A 2025 study of the leading commercial legal RAG tools found they hallucinated on 17 to 33 percent of queries (Magesh et al.).
What the research shows
Head-to-head comparisons on legal collections go back to the COLIEE shared task in 2021 and now cover U.S. case law, European and Asian statutes, bar exam questions, and end-to-end legal RAG. The pattern is consistent enough to state plainly. Every number below is read from the paper’s own results table; the papers are listed under Sources.
On case law, word matching holds its own
When no one has trained the embedding model on legal text, BM25 matches or beats it on every case-law benchmark published. The gap is widest on the hardest set, CLERC, built from 1.84 million U.S. opinions: BM25 found 48 percent of the cited cases within its first thousand results, the best general embedding model 42, and the late-interaction model ColBERT 18. In 2021 a team entered an untouched BM25 system in the COLIEE case-retrieval competition and placed second against every neural entry. By 2025, every team in that competition started from BM25 and added neural steps on top.
The one qualifier is important: this holds for embedding models as they come off the shelf. What happens when they’re trained on legal pairs is a separate result, below.
Table
| Benchmark | Metric | BM25 | Embeddings | Embedding model |
|---|---|---|---|---|
| CLERC | Recall@1000 | 48.3 | 42.4 | BGE En v1.5 |
| COLIEE (Zheng et al.) | Recall@10 | 38.1 | 32.7 | E5-large-v2 |
| CLERC (Zheng et al.) | Recall@10 | 11.8 | 6.8 | E5-large-v2 |
| LeCaRDv2 | Average recall | 58.7 | 50.0 | bge-m3 |
| COLIEE 2021 (DoSSIER) | Recall@100, paragraphs | 53.0 | 47.8 | DPR trained on COLIEE; 11.7 at whole-document level |
Sources: Hou et al. 2025, Zheng et al. 2025, LeCaRDv2 study 2026, Althammer et al. 2021.
On statutes asked in plain language, trained embeddings win
Flip the setting and the result flips. When the questions come from the public and the answers are statutes, the words rarely match, and an embedding model trained on question-and-statute pairs pulls far ahead. On BSARD, 1,100 questions from Belgian citizens against 22,600 articles of Belgian law, the trained model found 75 percent of the right articles in its first hundred results to BM25's 51. A 2026 Korean study of fire-safety regulations shows the same gap, and so does a U.S. benchmark of housing-law questions across fifty states.
Trained is the operative word. The same BSARD paper tried the embedding model before training it and it found 4 percent.
Table
| Benchmark | Metric | BM25 | Embeddings | Embedding model |
|---|---|---|---|---|
| BSARD | Recall@100 | 51.3 | 74.8 | Two-tower CamemBERT, trained on BSARD |
| Korean fire-safety statutes | Recall@10 | 23.4 | 53.8 | BGE-M3 |
| Housing Statute QA | Recall@10, lower bound | 18.3 | 24.4 | E5-large-v2 |
| BSARD, untrained | Recall@100 | 51.3 | 4.2 | CamemBERT, no training |
Sources: Louis and Spanakis 2022, Chae et al. 2026, Zheng et al. 2025.
Training on legal pairs flips the case-law result too
The CLERC authors took the embedding model that lost to BM25 and trained it on their own corpus of judges citing cases. Recall at a thousand went from 42 to 69, well past BM25. A separate 2025 study measured the effect on answers rather than on retrieval: with the same generator and the same legal corpus, a general embedding retriever produced answers as accurate as BM25, and a legally trained one was eight points better.
So the honest summary isn’t that embeddings fail on case law. It’s that embeddings earn their place only after someone trains them on the right pairs, and that general models, the ones most products ship with, don’t get there on their own.
Table
| Retriever | Recall@1000 | nDCG@10 |
|---|---|---|
| BM25 | 48.3 | 5.40 |
| BGE En v1.5, general | 42.4 | 3.43 |
| LegalBERT DPR, trained on CLERC | 68.5 | 14.67 |
Sources: Hou et al. 2025; answer accuracy from LRAGE 2025 (BM25 55.6, general DPR 55.1, legal-trained DPR 63.7 percent).
The hardest questions defeat both, and rewriting the question helps most
A 2025 Stanford benchmark built from bar exam fact patterns asked retrievers to find the passage a question turns on. BM25 found it in 5 percent of cases within ten results; the best general embedding model found it in 7. The authors' diagnosis was that these questions require reasoning before searching: you have to know what rule is at issue before any retriever can find it. When they had a language model restate each question in legal terms first, both retrievers improved, and the lexical one improved most.
A second 2025 study made the same point on a dataset of four million judicial citations. BM25 alone found the cited passage within ten results 28 percent of the time. An embedding model trained on the dataset managed 51. BM25 with the query rewritten by a trained language model reached 62.
“These findings suggest that retrievers must themselves be reasoners too.”Zheng et al., A Reasoning-Focused Legal Retrieval Benchmark, 2025
Table
| Benchmark | Query | BM25 | Embeddings | Embedding model |
|---|---|---|---|---|
| Bar Exam QA | As asked | 5.0 | 7.0 | E5-large-v2 |
| Bar Exam QA | Structured-reasoning expansion | 11.3 | 15.9 | E5-large-v2 (baseline plus the paper’s reported gains of 6.28 and 8.86) |
| LePaRD, 10K pool | As asked | 28.2 | 51.0 | DPR fine-tuned on LePaRD |
| LePaRD, 10K pool | GuRE rewriting | 62.2 | 60.2 | ModernBERT, untrained, with rewriting |
Sources: Zheng et al. 2025, Kim et al. 2025, Mahari et al. 2024.
Mixing both isn’t free
The natural compromise is a hybrid: run both, merge the lists. The studies that tried it got mixed results. The 2021 COLIEE team whose merged system beat BM25 on their development data lost to plain BM25 on the test data. A 2025 study of legal contracts and privacy policies added BM25 to an embedding pipeline, watched passage-level precision fall, and removed it. The Korean statute study found that rank fusion with BM25 lowered recall for both embedding models it tested. Where hybrids do help, in the COLIEE pipelines, they work by widening the pool of candidates rather than by blending scores.
Retrieval doesn’t cure hallucination
None of this changes what happens after retrieval. The Stanford study of commercial legal research tools tested products that retrieve before they answer and found hallucination rates between 17 and 33 percent, lower than a general chatbot’s but far from the zero some vendors claimed. The retriever decides what the model gets to read. Whether it reads correctly, and whether it cites what it read, is a separate problem that no index solves.
Why words win on case law
The results above aren’t an accident of the benchmarks. They follow from how courts write and how lawyers search.
The words are the law
A statute is a fixed string, and a court applying it quotes it. A standard adopted in one decision is repeated in the hundreds that follow it, in the same words, because changing the words would change the standard. “Particular social group,” “crime involving moral turpitude,” “exceptional and extremely unusual hardship,” “one central reason”: these phrases are the shared vocabulary of the question and the opinion. Matching them exactly is what you want, and an engine that treats them as approximate meanings loses information.
Citations and names are strings, not meanings
A large share of legal research is finding a specific thing: a reporter citation, a case name, a docket number, a subsection. These have no meaning to embed. They match or they don’t, and lexical search matches them.
Opinions are long, and the relevant part is a paragraph
The controlling paragraph of a forty-page opinion is a small fraction of its text. Lexical search scores each passage on its own and lets the best passage carry the document. Embedding search has to compress each chunk into one vector, and the authors of the largest U.S. benchmark attribute their models' weak results to exactly that: long documents and long queries that a single vector can’t hold.
General models learned the web, not the Board
An embedding model knows what “similar” means from its training text. Web text treats asylum and withholding as near-synonyms, and inadmissibility and deportability as the same idea. Legal research lives in those distinctions. Models trained on legal pairs recover, as the CLERC result shows, but that training needs a large labeled corpus and a full rebuild whenever the model changes.
The person or agent asking already speaks the vocabulary
The vocabulary gap is real, and the studies show where it bites: plain-language questions against statutes. But the gap is on the query side, and the query side now has a language model on it. An AI assistant restates “deported for drunk driving” as a question about convictions and removability before it searches. That’s the rewriting result above, and it’s why a lexical index behind a capable assistant performed like a trained embedding model, at a fraction of the cost and with its explanations intact.
You can check the match
When the result is a cited authority in a brief, the question “why did this come up” has to have an answer. A lexical hit answers it by pointing at the words. That transparency is worth more in law than in most fields, and it’s the thing embedding search gives up first.
How Passim AI searches
Passim is the library and the index. The assistant you already use does the reading and the reasoning.
- Your assistant writes the search. It turns your question into the terms decisions use. If a search comes back with weak matches, Passim says so in the result, and the assistant tries different words.
- Passim searches by words. Every collection, from precedent and unpublished decisions to statutes, regulations, and agency guidance, is a lexical index. Decisions are searched three ways at once: the whole decision, each passage, and the rule statements Passim has drawn from them, so a decision can be found by the paragraph that matters even when its caption gives nothing away.
- Authority counts. Supreme Court, Attorney General, and Board precedents are weighted so a landmark isn’t buried under the hundreds of later decisions that apply it. A decision that has been overruled or vacated keeps its place in the results but can’t be promoted as controlling.
- Citations resolve exactly, and history is a separate check. A citation lookup is an exact match, and a decision’s later treatment comes from ImmCite, Passim’s citator, as its own step rather than as a search score.
Why no embeddings, for now
We tested them. On a fixed set of searches where our lexical ranking had missed the right decision, a general embedding model recovered 2 of 43 and a legal embedding model 7 of 43. Reaching deeper into the lexical results recovered 8. That matches what the research says about case law, and it’s why Passim’s index is lexical today. If an embedding model trained on immigration decisions changes that, we’ll measure it the same way before it ships.
Sources
Every figure on this page is taken from the results table of the paper it’s attributed to. Benchmarks differ in corpus, query style, and metric, so compare rows within a chart, not across charts.
- Rosa et al. 2021. Yes, BM25 is a Strong Baseline for Legal Case Retrieval. COLIEE 2021.
- Althammer, Askari et al. 2021. DoSSIER at COLIEE 2021: dense retrieval and summarization-based re-ranking for case law retrieval.
- Louis and Spanakis 2022. A Statutory Article Retrieval Dataset in French. ACL 2022.
- Mahari et al. 2024. LePaRD: A Large-Scale Dataset of Judicial Citations to Precedent. ACL 2024.
- Hou et al. 2025. CLERC: A Dataset for U.S. Legal Case Retrieval and Retrieval-Augmented Analysis Generation. NAACL Findings 2025.
- Zheng et al. 2025. A Reasoning-Focused Legal Retrieval Benchmark. CS&Law 2025.
- Kim et al. 2025. GuRE: Generative Query Rewriter for Legal Passage Retrieval. NLLP 2025.
- LRAGE 2025. Legal Retrieval Augmented Generation Evaluation Tool.
- Reuter et al. 2025. Towards Reliable Retrieval in RAG Systems for Large Legal Datasets.
- Pipitone and Houir Alami 2024. LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain.
- Chae et al. 2026. Beyond Case Law: Evaluating Structure-Aware Retrieval and Safety in Statute-Centric Legal QA.
- LeCaRDv2 study 2026. When Rules Learn: A Self-Evolving Agent for Legal Case Retrieval.
- COLIEE 2025 overview. An Overview of the COLIEE 2025 Competition.
- Magesh et al. 2025. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies.
- Blair and Maron 1985. An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System. Communications of the ACM. The original warning about keyword search: searchers believed they had found 75 percent of the relevant documents and had found 20. Summarized in Richmond Journal of Law and Technology.