We’ve been building Parsewise for close to two years now and we’ve always understood the tradeoffs between different search methods empirically. This is our attempt to formalise that understanding.
TLDR: search methods for candidate retrieval charted - click for details
These are generalised scores. Different use cases would score differently: looking up exact invoice numbers is not the same as looking for customer sentiment.
* The price and speed of LLM based methods is improving quickly.
Foreword
We will separate out retrieval and reranking because they’re largely composable. No matter how the candidates are sourced, we can use various metrics to prioritize them.
For candidate retrieval, there are two key events: ingestion and querying. The methods we’ll cover optimise based on the usage patterns, whether there are many repeating queries, whether the corpus is big or small, static or dynamic. We’ll see that often there is an intermediary layer being built at ingestion time. We can think of it as an index, whether it’s keyword dictionaries, semantic embeddings or entity graphs.
While Parsewise helps build a structured data ontology, here we will be zooming in on a single data point. (An interesting observation here is that data transformation (or ETL) via LLMs can be thought of as a series of search queries.) For a broader look at document transformation pipelines, see here.
Now for the methods themselves.
1. Keyword based
The simplest one. Look for “apple” and you’ll get back everything about the iPhone maker as well as the fruit; it won’t match “the forbidden fruit that Eve bit” nor “🍎”.
By default it is restrictive, matching exact sequences of characters while disregarding meaning.
There are a variety of techniques to expand the match scope. Syntactically (incl. stemming, fuzzy, phonetic, grep, regex) and semantically (synonyms, translations, LLM based expansion).
- Guarantees
- Exact matches are all guaranteed and cannot be missed
- Good for
- Technical term lookups (e.g. medicine, chemicals)
- Challenges
- Cannot by itself support logic in the query, conditionals, semantic abstractions
2. Semantic search
Now instead of just looking for syntax and characters, we rely on meaning. This is most commonly achieved by embedding document chunks into a high dimensional space.
If you look for “Apple financials” you’ll get back the financial statements of the company, and maybe the numbers for the international apple trade. The embedding lacks granular detail to distinguish between the two.
With query expansion or HyDE, the query may become something like “Apple’s revenue and capex numbers were [...]”. That will be more similar to the financial statements, but will equally look at old, out of date statements.
The challenge then becomes looking for something specific like “Apple’s revenue for AirPods in 2024”, as this will match semantically both documents around AirPods, and financial statements from multiple years that may mention 2024.
Semantic search used to be most commonly implemented with RAG. It works on the basis of taking chunks and embedding them in a vector space at ingestion time (dense passage retrieval). That means that querying is a fast db lookup in a vector space, usually via approximate nearest-neighbour search.
There are various techniques around chunking in different sizes or semantically, training custom embedding models, hypothetical embeddings (HyDE) and many more to improve performance.
It can also be combined with keyword search before reranking for improved guarantees.
- Guarantees
- None; the query embedding and its similarity are nondeterministic
- Good for
- Large static corpus with lots of queries
- Challenges
- Domain specific terminology and document chunks collapse in a small vector space with generic embedding models, making retrieval difficult and noisy
3. Graph and relationship based search
This approach is most helpful if we have a query like “what are the top 3 country risks for Apple’s supply chain”. Then the graph can be traversed to find the relevant pieces of information.
This technique is more use case specific, in that it assumes entities and relationships in the corpus that we can build into a graph at ingestion time. GraphRAG is the recent form of this: an entity graph plus community summaries, used for both local entity questions and corpus-level ones.
Then at query time the graph itself can be traversed.
- Guarantees
- Relationship based queries can deterministically guarantee no false negatives and no false positives
- Good for
- Inherently entity relationship based queries such as financial transactions, ownership structures
- Challenges
- Any query relying on details not modelled by the graph
- China31%
- Taiwan21%
- Vietnam17%
Ranked by share of spend routed through each country
No edge carries labour audits, so there is nothing to walk. The graph can only answer what it was built to model.
4. Agentic
This is essentially asking Claude or ChatGPT to search through files. The model drives the exploration and chooses the tools to use, in the ReAct pattern of alternating search and reasoning. The common patterns often include using glob to find relevant folder or file names and grep to do keyword searches. Based on the results the LLM can decide whether to stop and respond or keep expanding the search (e.g. by using different keywords, reading a table of contents etc.).
Recall is bounded by the keywords the model happens to pick: a page that has the answer in different words may not be returned.
A related shortcut is to send the entire corpus straight to a long-context LLM. Convenient when everything fits, but not reliable and runs into scale limits.
- Guarantees
- None; results depend on the model and harness
- Good for
- Ad-hoc exploratory queries where 100% correctness is not required
- Challenges
- It is hard for the user to build confidence in the results
Not one line on this page says AirPods, so grep never returned it and the model never opened it, even though the number is on it.
Whatever the keyword missed, the model never saw.
4.1. Skills / harnesses
Skills and harnesses provide some degree of programmability for LLMs, closer to query decomposition than to an unconstrained agent loop. For example, we could explicitly ask for a subagent to be spawned for every single page in the corpus. With each of them storing their results and provenance we could build a visualization for users to check results.
Complexity arises in various areas:
- Providing the right amount of context (e.g. a page may be missing headers for a table); giving the subagent permission to explore leads to false positives
- Working with subagents at scale (a sizeable corpus requires thousands to millions of LLM requests in parallel; managing the memory and storage substrate becomes complex)
- Errors, failures, retries
- Productionizing both the view and storage layer such that the users can rely on the same UX
- Guarantees
- Exhaustive search
- Good for
- Structured extraction where missing a result is problematic
- Challenges
- Can be noisy with false positives; large corpora with many queries are slow & expensive; complexity of building, maintaining
5. Parsewise
Parsewise is closest to a harness. It works by having an LLM look at every page individually for a given query, along with additional context and metadata.
That means that even pages where keywords do not appear explicitly, but that contain the relevant information are surfaced.
Then all candidate results become visible to users and they get a combined / resolved value as well. This enables Parsewise to both achieve state-of-the-art benchmark results, and to allow subject matter experts to quickly verify results.
Additionally, we have an agent (called Navi) that routes between ad-hoc queries and exhaustive search.
- Guarantees
- Exhaustive search
- Good for
- Structured extraction where missing a result is problematic
- Challenges
- Large corpora with many queries are slow & expensive
Read like every other page, so the row that carries the number counts whether or not it ever says AirPods.
Three pages carry the answer, and one of them no keyword search would return.
Filtering & Reranking
Regardless of the retrieval method, we can narrow down the candidates and sort them. Filtering tends to come first because it can reduce the search scope and thus improve on speed and cost. Reranking can help either further narrow down, or have the LLM or human attention focus on the most relevant candidates. Think of the Google search results page (in the pre-AI era), and how most people stop at the top few results.
Metadata, tags, folders, etc. enable deterministic filtering, but require setting up a rich data catalog in the first place.
Authority, freshness, document type, PageRank, TF-IDF and BM25 provide deterministic rankings, but tend to require use case specific tuning.
An agentic or LLM pass (cross-encoder reranking) can filter and rerank semantically.
Challenges
There are a few challenges worth calling out that make some methods more suitable than others.
Logic
'And', 'or', 'if', 'else' are contained in user queries. “Give me the revenues for the last 5 years” can be interpreted as a sequence of logical operations. Such queries are challengin for simple keyword and semantic search.
Multiple results
Like in the example above, queries often span multiple results, so we can’t always narrow down to the single best candidate. This makes filtering & reranking particularly challenging.
All candidates
When the user is looking for all candidates, it's particularly challenging to define the correct threshold for vector based methods. Set the threshold low and you miss results, set it high and you get many irrelevant ones.
Target schema
Data transformation (ETL) benefits from having a predefined target schema, however, in practice this frequently changes with business requirements. Thus maintenance of use case specific methods is expensive.
Generalisability
Generalisability matters when users work across different corpora. In this case, creating deterministic synonym dictionaries, custom embedders, and learned rankers are not practical.
File types
File types often require distinct strategies for some methods. E.g. searching across PDFs is different from excels and as such both keyword and vector based methods require adaptation.
Document contents
Document contents can carry information that is not directly available to search via keywords or vector similarity (think images, tables, diagrams, layouts).
Conclusion
Decades of research into search methodologies have resulted in a rich ecosystem of methods, that each carry trade-offs. LLMs and agents offer a very interesting new capability that is able to combine the very best of these, but raises new challenges around verifiability. This is the challenge that we’ve built Parsewise for, and what enabled our SOTA results on grounded reasoning.