Navi is now state of the art on Databricks’ OfficeQA Pro V2 See the results

How we reached State-of-the-Art on OfficeQA Pro V2

Getting more from the same model

Nik Bozhinov

Abstract illustration for the OfficeQA Pro V2 article

We achieved state-of-the-art results on Databricks' OfficeQA Pro V2 benchmark. Navi answered 73.3% of the questions correctly, 10 points ahead of the next best result. It cost $2.24 per question, less than Claude Code working from the raw PDFs.

This builds on our state-of-the-art result on the first version of OfficeQA. In this post, we go through what made the difference this time and how we approach grounded reasoning at scale (see also Navi at scale).

One note before we start: we ran this evaluation with Navi alone, without Parsewise agents, which search every page of the corpus. We'll cover a run with agents in a follow-up post.

Accuracy against cost per question on OfficeQA Pro V2, with Navi ahead of every other result
Accuracy against cost per question on the 90 OfficeQA Pro V2 questions. We ran Navi and both Claude Opus 5.5 baselines in November 2026. All other results are as published by Databricks.

Harder questions, rougher scans

OfficeQA Pro V2 covers more than two centuries of US federal accounts, close to 120,000 pages in all. Much of it is old, scanned and packed with dense financial tables, like the pages below. As in the first version, the same figure often shows up in several documents with different values, and gets revised over time. That's very close to what we see in our customers' data, and it's why we keep coming back to this benchmark as it evolves.

What makes V2 harder than its predecessor is mostly the questions. In the first version, a question usually needed a couple of documents. Now a typical question pulls from five or six, and the hardest one needs more than twenty. Only a small part of the corpus is ever relevant, so before you can reason about anything, you have to find the right handful of documents in a corpus full of annual reports that look much alike. The scans also look worse overall than in the first version, so even reading the numbers correctly turned out to be a challenge.

Some of the hardest pages to read (yes, this is the actual resolution)

Scanned page from 1941: Faint figures spread across a wide table

1941 Faint figures spread across a wide table

1 / 9

Baseline: What if we just used a frontier model?

Before running Navi, we wanted to know what the current frontier looked like. Models have improved a lot since Databricks published the benchmark, and Navi uses Opus 5.5 High as its reasoning model, so the obvious question was how far Opus gets on its own. We ran it in Claude Code on the raw PDFs, and it scored 62%, close to the best published result. That didn't surprise us. Opus handled the raw PDFs well and was good at finding the right source documents, and the jump in model capability did the rest.

What did surprise us was how long some questions took. Opus spent a lot of time reading scanned pages and checking them, and the longest question took over an hour, which pushed costs up too. We've long believed that giving a model a parsed text layer makes a big difference to speed and cost, and this was a good chance to test it.

We ran the same setup again, but this time gave Claude Code the corpus as parsed by Parsewise, the same text Navi works from. The typical question took a third of the time and cost less than half as much. Accuracy barely moved, from 62% to 63%. We think that's because Opus was already good at finding the right documents, so the text layer mostly saved it work.

Same model, raw PDFs against parsed text

Raw PDFsParsed text
Accuracy62%63%About the same
Typical time per question7 min2.3 min3× faster
Longest question65 min10.5 min6× faster
Cost per question$2.63$1.1257% less

Claude Opus 5.5 High in Claude Code on all 90 OfficeQA Pro V2 questions, November 2026.

Use a strong PDF parsing model

As you have seen above, pre-parsing the text makes processing a lot more efficient for your harness. When asked to find the "Total Patent Office receipts for March 1908", your AI can now quickly look through the extracted text. The alternative is finding the scanned document for 1908 and then parsing its pages.

You might already see how a revised figure published five years later can easily be skipped. Having the corpus as searchable text makes your agents more efficient, and it also makes those later revisions findable. Whether the agent actually uses them is down to the harness, which we cover below.

An 1873 scan and the text we extracted from it

Scan

Scanned 1873 table of custom-house appropriations, printed small with a smudged first line

Extracted text

RECEIPTS AND EXPENDITURES.      469

Custom-house, [Portland, Oreg.] (1874)4178,500 00......................78,500 0028,225 00...........50,275 00
Custom-house, Machias, Me.382,000 003,501 65...........5,501 655,501 65......................
Custom-house, Portland, Me............4,851 311,251 126,102 432,267 05...........3,835 38
Custom-house, Chicago, Ill............1,997,547 4766,365 712,063,913 181,556,619 60...........507,293 58
Custom-house, Chicago, Ill. (1874)41800,000 00......................800,000 00......................800,000 00
Custom-house, Newport, R. I............2,141 75...........2,141 752,141 75......................
Appraiser's stores, Philadelphia, Pa............4,877 90...........4,877 904,866 70...........11 20
Appraiser's stores, San Francisco, Cal.41408,000 00......................408,000 0020,000 00...........388,000 00
Custom-house, &c., Fall River, Mass.39200,000 00......................200,000 00...........200,000 00...........
Custom-house, &c., Fall River, Mass. (1874)41200,000 00......................200,000 00103,200 00...........96,800 00
Custom-house, &c., Boston, Mass. (1874)4130,000 00......................30,000 00......................30,000 00
Custom-house, &c., Hartford, Conn.38, 39400,000 00......................400,000 001,129 80300,000 0098,870 20
Custom-house, &c., Cincinnati, Ohio38, 391,000,000 00......................1,000,000 00417 30370,000 00629,582 70
Custom-house, &c., Cincinnati, Ohio (1874)41750,000 00......................750,000 00......................750,000 00
Custom-house, &c., Charleston, S. C............90,000 00...........90,000 0090,000 00......................
Custom-house, &c., Charleston, S. C. (1874)41172,081 52......................172,081 5279,841 58...........92,239 94
Custom-house, &c., Memphis, Tenn.4125,000 00......................25,000 00......................25,000 00
Custom-house, &c., Nashville, Tenn. (1874)41150,000 00......................150,000 00......................150,000 00
Custom-house, &c., Saint Louis, Mo.39500,000 00300,000 00...........800,000 00309,098 42...........490,901 58
Custom-house, &c., Saint Louis, Mo. (1874)411,000,000 00......................1,000,000 009 13...........999,990 87
Custom-house, &c., Albany, N. Y.38, 39450,000 00......................450,000 0025 30350,000 0099,974 70
Custom-house, &c., Albany, N. Y. (1874)41150,000 00......................150,000 00......................150,000 00
Custom-house, &c., Ogdensburg, N. Y.383,000 00......................3,000 00......................3,000 00
Custom-house, &c., New Orleans, La............265,000 00...........265,000 00265,000 00......................
Custom-house, &c., New Orleans, La. (1874)41170,000 00...........16,942 75186,942 75185,938 75...........1,004 00
Custom-house, &c., Bath, Me.455,000 00......................5,000 00......................5,000 00
Custom-house, &c., Rockland, Me.3925,000 00......................25,000 0012,000 00...........13,000 00
Custom-house, &c., Rockland, Me. (1874)4175,000 00......................75,000 00......................75,000 00
Custom-house, &c., Baltimore, Md.3850,000 00......................50,000 0050,000 00......................
Custom-house, &c., Port Huron, Mich.39200,000 00......................200,000 005,000 00...........195,000 00
Custom-house, &c., Port Huron, Mich. (1874)41100,000 00......................100,000 00......................100,000 00
Custom-house, &c., Evansville, Ind.41100,000 00......................100,000 0045,356 30...........54,643 70
Marine hospital, Chicago, Ill. (1874)4123,825 44......................23,825 4423,825 44......................
Marine hospital, San Francisco, Cal.4558,789 56......................58,789 56......................58,789 56
Repayments to importers for unascertained duties473,120,192 10373 6563,784 603,184,350 353,183,394 37...........955 98
Payment of debentures, drawbacks, bounties, or allowances471,176,055 18...........24,694 001,200,749 181,200,749 18......................
Distributive shares of fines, penalties, and forfeitures47626,176 742,242 686,050 51634,469 93634,389 93...........80 00
Refunding duties, sec. 4, act March 3, 187147134,552 06......................134,552 06134,312 66...........239 40
Drawbacks on certain articles imported into Chicago47192,155 95...........11,199 77203,355 72203,355 72......................
Unclaimed merchandise471,440 8079 86...........1,520 661,520 66......................
Refunding duties to extend the warehousing system47935 24......................935 24935 24......................
Debentures and other charges4735,655 81109 19782 1036,547 1036,546 03...........1 07
Compensation of persons employed in insurrectionary States47710 02......................710 02710 02......................
Additional compensation to collectors47319 53......................319 53319 53......................
Refunding duties on tea and coffee47257,231 82......................257,231 82257,231 82......................
Relief of James F. Joy4847,776 40......................47,776 4047,776 40......................
Refunding to James O. P. Burnside proceeds of cotton seized465,284 00......................5,284 005,284 00......................

All 47 extracted rows balance: appropriations, earlier balances and repayments add up to the total available, and so do payments, the amount carried to surplus and the closing balance.

One thing to note is that the quality of the parsing is closely tied to how well you will answer your questions. On scanned pages where the numbers can barely be made out, incorrect parsing can throw off your model, and a single misread digit leads to a wrong answer that looks right.

At Parsewise, we have been evaluating and optimising this pre-processing for a long time, across a few dimensions. First, it needs to be quick, since Navi and Parsewise agents can't start reasoning until the text is parsed. Second, it needs the highest possible accuracy, so that your figures match up even on the toughest scans. Finally, we have to balance both against the cost of the model.

A dedicated harness unlocks SOTA accuracy

By this point you might be wondering where the extra 10 points of accuracy come from. It isn't the extracted text on its own, since that didn't change Opus 5.5 High's accuracy in Claude Code. As you will see below, it's the harness: with the right context and guidance, a data-centric agent still beats the underlying model by a wide margin.

One way Navi does that is by understanding the underlying data much better. Even with our preferred parsing model, we found that data on some scanned pages was close to unreadable. When answering a question, Navi keeps track of the figures it uses in its calculations and checks the ones that don't add up. To do that, it can zoom into the page image itself or ask a stronger model to re-read the figures on that page. It can also look at the surrounding pages, totals and neighbouring years to work out what the value must be.

Checking a figure that doesn't add up

The Carried forward row of a scanned 1873 table
73,521,532.0321,437,792.96+ 1,791,570.2296,750,895.21
≠
Printed total96,550,895.21
200,000.00 out

This is also where the revised figure published five years later comes back in. Searchable text makes it findable, but Navi still has to go looking for it. So Navi understands the scope of the corpus and is tasked with keeping a good balance between exploration and exploitation. It won't automatically pick the first data source it finds. Instead, it actively looks for revisions and other values that might contradict what it found, and only answers the question once it is confident in its findings.

Looking past the first answer

"What were the Patent Office receipts for March 1908?"

  1. 1908
  2. 1909
  3. 1910
  4. 1911
  5. 1912
  6. 1913

Answers $191,075, the revised figure

What we also observed is that answering the question once you have all the data points is a lot less straightforward than it might seem. That's less surprising when you consider that some questions in the benchmark carry an underlying ambiguity. Take a question like "How much did the government spend on the Patent Office in the 1920s?". An analyst might mean the calendar years 1920 to 1929 or the fiscal years that cover them, the figures as first published or as later revised, and gross spending or spending net of the office's own receipts.

To handle this, Navi approaches the question as an analyst would: before calculating anything, it works out what the question is really asking and what the numbers mean. If web search is enabled, it may even look online to get more context on the question. Finally, when a question allows more than one reading, it picks the most likely one and states the alternative alongside its answer.

Working out what the question means

"How much did the government spend on the Patent Office in the 1920s?"

Which years?Which figures?Gross or net?
  1. Which years?Treasury reports run by fiscal year.
  2. Which figures?Revisions win unless the question says otherwise.
  3. Gross or net?Spending means what the office paid out.

A hypothetical question with illustrative totals.

Is there a place for a human in the loop?

This ambiguity raises an interesting question. When colleagues work together, not every request is a one-shot question with a single clear answer. By talking it through and considering alternatives, you build a much better picture of what the question means and why it was asked. We think working with AI should be no different.

To test how close agents already get on their own, we allowed Navi to provide up to 5 alternative answers based on different question interpretations. This raised accuracy by about 6 more points. It would be interesting to explore a version of this benchmark where a human could steer the agent to reduce this uncertainty further.

Steering Navi to the right reading

How much did the government spend on the Patent Office in the 1920s?

$29.4M

Fiscal years, revised figures, gross spending

Treasury reports run by fiscal year, so I used those. Did you mean calendar years?

Yes, calendar years

$30.1M

Calendar years, revised figures, gross spending

What comes next

As we mentioned at the start, this run used Navi alone. Navi works like an analyst on top of a project's processed data: it explores the documents, reasons across them and runs the calculations. We have spent over a year getting it to where it is now.

On a corpus this large, Navi doesn't read every page. It explores the data and reasons over it, so it can miss small details. That's what Parsewise agents are for. Users or Navi create them, and each agent runs a deep search over the whole corpus, page by page. Agents are one of the foundations of the platform, and they guarantee users complete traceability over the data.

In a follow-up post, we will publish the results of a run with Navi and agents together. We expect it to raise accuracy further.

Share this post