We asked Ardoq AI 200 two-hop questions of the form "Who are the owners of the applications that realize the 'capability' business capability?": 100 where the applications have owners and 100 where they have none. The questions went to the Ardoq Assistant with Claude Opus 4.6, Sonnet 4.6, Opus 5.5 and Sonnet 5.5, between 15 and 19 2026. Of 4,000 questions asked, 3,968 are scored; 32 that failed because of test-infrastructure issues (e.g., tool exhaustion or timeouts) are left out. Goals, method and research questions are on Retrieval Benchmarks
Summary of results
Quality
When owners exist, Ardoq AI usually finds all of them. Opus 5.5 answered 100% correctly in all 5 combinations of retrieval method and asset size, Opus 4.6 99–100%, Sonnet 5.5 97–100% and Sonnet 4.6 92–99%. Wrong answers were near misses: the partial-credit score stayed at 0.987 or higher in every combination.
A wrong answer usually leaves an owner out. Of the 24 wrong answers where owners exist, 21 left out at least one expected owner and 10 named someone who is not an owner (one answer can do both). Most of the owners left out are teams rather than people. Sonnet 4.6 gave 18 of the 24.
When no owner exists, the newer models always say so. Opus 5.5 and Sonnet 5.5 were 100% correct in all 10 of their combinations. 15 answers named people who are not owners of the capability’s applications: 11 from Sonnet 4.6 and 4 from Opus 4.6.
20 questions did not finish, all with Sonnet 4.6. 5 repeated the same calls until they reached the step limit; 15 reached the time limit.
Consistency is not measured yet. Each question was asked once.
How to read it. Each bar is one model, with every retrieval method and asset size combined: 500 questions per bar. Green is the share answered correctly, red the share answered wrongly, and purple the share that did not finish (the AI ran out of steps or time before answering). The left panel covers questions where owners exist, the right panel questions where none do.
What it shows. Opus 5.5 is correct on every question. Sonnet 4.6 falls short on both kinds of question; wrong answers outnumber unfinished questions.
How to read it. Each row is one combination of model, retrieval method and asset size. The dot is the share of about 100 questions answered correctly; questions that did not finish are left out of this share and counted in the next chart. The line is the 95% range: because we tested 100 questions rather than every possible one, the true rate could plausibly lie anywhere along the line. Even 100 out of 100 correct gives a range of 96–100%. Where two lines overlap, the difference could be chance; where they don't, it is real.
What it shows. Almost every line overlaps near 100%. Sonnet 4.6 with neighbors sits lowest where owners exist (92%), the one combination measurably below the best. Measurably below means the top of a combination’s 95% range is below the bottom of the best combination’s range for the same question kind.
How to read it. Each bar counts the failed questions, out of 100, for every combination with at least one failure, for both question kinds. Red is a wrong answer; purple is a question that did not finish because the AI reached its step limit or time limit.
What it shows. Failures are concentrated in Sonnet 4.6, especially on broad viewpoints. The other models' failures are single wrong answers.
How to read it. Two-hop answers are lists of owners, so an answer can be partly right. For questions where owners exist, the left panel counts answers that left out at least one expected owner; the right panel counts answers that named someone who is not an owner. One answer can do both, so the two panels do not add up to the number of wrong answers.
What it shows. Leaving an owner out is the more common mistake. Opus 5.5 made neither.
Latency
A typical question takes 15–33 seconds when owners exist, and 17–52 seconds when none do (median per combination).
Slow questions are concentrated in questions with no owner, on reports. 123 of 3,948 answered questions took more than 60 seconds: 119 had no owner and 103 used reports. None were Opus 5.5.
Opus 5.5 is faster than Opus 4.6 on the same questions: a median 14% less time when owners exist and 32% less when none do. It was faster in 8 of 10 combinations; the exceptions are viewpoints where owners exist (about the same on broad viewpoints, 32% slower on curated ones).
The newer models stay closer to the minimum number of calls. When owners exist, Opus 5.5 and Sonnet 5.5 make 0.2–0.8 extra retrieval calls on average, against about 2 for the 4.6 models. When no owner exists: 1.2–1.7 extra, against 4.2 (Opus 4.6) and 6.6 (Sonnet 4.6).
How to read it. Each bar is a typical time per question for one model and retrieval method: the median, averaged over asset sizes and both question kinds.
What it shows. Sonnet 5.5 is quickest on neighbors and viewpoints; Opus 4.6 is slowest on every method.
How to read it. Each bar is the median time per question: half the questions were faster, half slower. The black tick is the 90th percentile: 9 in 10 questions were faster than this, so the gap between bar and tick shows how long the slow ones take. Colors are models; rows are retrieval methods and asset sizes. The top panel is questions where owners exist, the bottom panel questions where none do.
What it shows. Questions with no owner take longer, and their slowest questions take much longer on reports with Sonnet 4.6 and Opus 4.6.
How to read it. A tool call is one request the AI makes to Ardoq. A retrieval call is one that fetches or finds data; metamodel lookups and link-building calls are not counted. Each bar is the average number of retrieval calls per question; the black tick is the minimum a question needs: 3 for neighbors (find the capability, read its neighbors, read its applications' neighbors) and 4 for reports and viewpoints (find the capability, list the assets, read a definition, get the data). Bars past the tick are extra calls; bars short of it mean a step was skipped. Same layout as the time chart.
What it shows. Opus 5.5 and Sonnet 5.5 sit at or near the minimum. On viewpoints most models stay below it, skipping the definition read. Reports take extra calls almost everywhere, and many more on questions with no owner.
Cost indication (tokens)
Answering context is 17,000–71,000 tokens (median per combination), more than single-hop needed because two hops bring back more data. Broad viewpoints need the most: 55,000–71,000. The largest single answering context was 340,000 tokens.
Tokens per question range from about 67,000 to 960,000 (mean per combination), highest on report questions with no owner and Sonnet 4.6.
Average tokens per question by model: Opus 5.5 171,000; Opus 4.6 180,000; Sonnet 5.5 243,000; Sonnet 4.6 350,000.
Ardoq AI reads only what it needs from reports. The reports hold about 470,000 to 5.3 million tokens; each report call returned a median of 320–1,240 tokens. Viewpoint calls return the traversal for the capability asked about: 3,000–4,000 tokens on curated viewpoints, 26,000–41,000 on broad ones.
How to read it. For report questions: blue is the size of the whole report, green what the model held when it answered, both as the median per question. The scale is logarithmic: each gridline is ten times the one before, so a bar one gridline longer is ten times larger.
What it shows. A broad report holds about 5 million tokens and a curated one about half a million, yet both are answered from roughly 35,000 tokens, because the AI asks Ardoq for the matching rows only.
How to read it. A token is roughly three-quarters of a word: the unit AI models read and are billed in. Answering context is how much the model was holding (the question, its instructions and everything retrieved so far) when it answered. Bars are the median per question, in thousands of tokens, with both question kinds combined.
What it shows. Broad viewpoints stand out at 55,000–71,000 tokens; everything else stays between 17,000 and 45,000.
How to read it. Blue is what the AI could have read: the whole report, or for viewpoints what the viewpoint calls returned (a viewpoint runs from the capability the AI passes, so there is no larger whole to read). Orange is what the last report or viewpoint call returned; green is what the model held when it answered. Log scale, as in the previous chart.
What it shows. Report calls return a tiny slice of the report. Viewpoint calls return their whole traversal, so a broad viewpoint sends ten times more data per call than a curated one, which is why broad viewpoints need the most context.
How to read it. Each bar is the total tokens per question, summed over every request the AI made. Each tool call adds to the conversation the model re-reads, so this grows with the number of calls. Same layout as the time chart. The Assistant reuses cached prompt tokens, so its counts overstate what it is billed.
What it shows. Tokens follow tool calls: Sonnet 4.6 uses by far the most on questions with no owner, where it makes the most calls.
What we tested
Questions: "Who are the owners of the applications that realize the 'capability' business capability?" 100 capabilities whose applications have owners (the expected answer is the full set of owners, often several) and 100 whose applications have none (the expected answer is no owner). Expected answers come from the data itself, not written by hand.
Access: the Ardoq Assistant only. This benchmark has no MCP runs.
Retrieval methods: neighbors (walk the graph from the capability), reports, and viewpoints.
Asset size: each report and viewpoint question is asked against a curated asset (narrow and sparse) and a broad, dense one, with the same question and expected answer.
Models: Claude Opus 4.6, Sonnet 4.6, Opus 5.5 and Sonnet 5.5, at the Assistant's default settings.
Scoring: strict. The names returned must match the expected names exactly; one missing or one extra name fails the question. A partial-credit score (F1) is also reported, because two-hop answers are lists. Correct-answer rates carry 95% ranges.
Repeats: each question was asked once per configuration.
Corrections after tracing failed questions: 32 questions that failed because of a test-infrastructure error are left out of every rate. 59 Sonnet 5.5 answers that the evaluation endpoint rejected after an invented tool call were scored from the answer recorded in the trace; all 59 were correct. Questions that looped or timed out stay in as "did not finish".
Discussion by research question
RQ1: How reliable is Ardoq AI at retrieval? This covers accuracy (how often the answer is right) and consistency (whether the same question gets the same answer when asked again), on questions that have an answer and on questions that do not.
Reliable with the newer models; noticeably less so with Sonnet 4.6.
Where owners exist, correct answers range from 92% to 100%. 1 of 20 combinations is measurably below the best: Sonnet 4.6 with neighbors (92%).
Where no owner exists, all combinations score 95–100% and cannot be told apart at this sample size.
Wrong answers are near misses (partial-credit score at least 0.987), most often a missing owner rather than an invented one.
Consistency is not measured: each question was asked once.
Likely causes of wrong answers
Every name in a wrong answer, and every owner left out, was compared with the graph around the question.
Teams left out: of the 23 owners left out where owners exist, 20 are teams (organizational units), although teams are only 16 of the 358 expected owners. Sonnet 4.6 left out only teams. On questions with a team owner, Sonnet 4.6 is correct on 78% against 99% without.
Long lists on report questions: where no owner exists, 15 answers named 208 people; 4 answers on reports hold 171 of them. These names do not belong to any structure near the capability. Reading the wrong rows of a report is a likely cause, but the traces have not been checked.
Experts of the applications: 31 extra names, all from Sonnet 4.6.
Owners via a child capability: 4 extra names where owners exist.
Similarly named capabilities share a base name in only 7 cases and were not a source of errors.
In general graph terms, the main structural error is answer-type narrowing: the AI treats only one of the types the metamodel allows as an answer (people, but not teams, as owners). The long lists resemble those seen on three-hop report questions.
RQ2: Does the size of the Ardoq asset the AI retrieves (viewpoint, report, etc.), and therefore the context size, affect retrieval accuracy?
Not measurably at this sample size, but broad viewpoints show small drops and need much more context.
On reports, accuracy is unchanged or within one question. On broad viewpoints, questions with owners drop by 1 point (Opus 4.6), 3 points (Sonnet 5.5) and 4 points (Sonnet 4.6); Opus 5.5 does not drop. These drops are within the 95% ranges.
Answering context rises sharply on broad viewpoints: +170% to +256% where owners exist and +85% to +140% where none do. On reports the change is −5% to +49%.
Time per question changes by −22% to +71%, mostly upward.
How to read it. Each dot compares the same questions on a curated asset and a broad one, for one combination of model, method and question kind. Left: change in correct answers, in percentage points. Middle: change in answering context, in percent. Right: change in time, in percent. The vertical line is no change; dots to the right mean the broad asset scored higher, used more tokens or took longer.
What it shows. Correct answers stay on or near the line. Context moves far to the right for viewpoints, and time mostly moves right.
RQ3: Is it beneficial to curate assets to get more reliable retrieval? A curated asset is narrow and sparse: its information content is specific to the retrieval question.
Yes for viewpoints: curated viewpoints need roughly half to a third of the context, with accuracy equal or slightly better.
The reason is the data per call: a curated viewpoint call returns 3,000–4,000 tokens, a broad one 26,000–41,000.
For reports, the benefit is smaller: the AI filters both kinds of report down to a small slice, so context and time differ less between curated and broad.
RQ4: Is answer quality different between Assistant and MCP access to the same tools?
Not tested in this benchmark: only the Ardoq Assistant was run. The single-hop results compare the Assistant and MCP.
Latency compared with the expected minimum
Neighbors: Opus 5.5 used exactly the minimum of 3 retrieval calls on every question and Sonnet 5.5 nearly so (3.04). The 4.6 models used about 4: one extra data call.
Reports: above the minimum everywhere where owners exist (4.6–8.8 calls). On broad reports, Opus 4.6, Sonnet 4.6 and Opus 5.5 looked up applications by name one at a time, adding 3–5 extra lookups per question. Sonnet 5.5 stayed at 5.2–5.5 calls, with one extra data call.
Viewpoints: usually below the minimum, at about 3 calls: the AI skipped reading the viewpoint definition and still answered correctly. Exceptions are Opus 4.6 on broad viewpoints (4.8 calls, extra definition reads) and Sonnet 4.6 on curated viewpoints (5.6 calls, extra asset selection).
Questions with no owner take more calls everywhere; Opus 4.6 and Sonnet 4.6 reach 14–16 calls on reports, while Opus 5.5 stays at about 7.
Token cost in more detail
Two hops need more answering context than single-hop (median 17,000–71,000 tokens, against 10,000–44,000 for single-hop), but it stays far below what these models can hold.
Tokens per question follow the number of calls. Opus 5.5 uses 31% more tokens than Opus 4.6 where owners exist (it holds more context per call) but 25% fewer where none do (it makes fewer calls).
The Assistant reuses cached prompt tokens, so its billed cost is lower than its token counts suggest. Token counts are shown, not prices.
4. Hypotheses: what the evidence says
Hypothesis | Verdict | Why |
H1: accuracy holds near ceiling up to roughly 256k tokens of answering context and falls visibly beyond roughly 500k | Not testable yet | The median answering context was 17,000–71,000 tokens in all 20 configurations; one single answering context reached 340,000. A single case cannot show a trend. |
H2: neighbors needs fewer tool calls than reports or viewpoints | Supported | Stated for single-hop. For two-hop, neighbors used fewer retrieval calls in 30 of 32 comparisons (same model and question kind, against each report and viewpoint asset): all 16 against reports and 14 of 16 against viewpoints. The exceptions, both where owners exist: Sonnet 4.6 on broad viewpoints and Opus 4.6 on curated viewpoints (3.1 calls against 4.0), which skipped the definition read. The margin over viewpoints is often small: the 5.5 models used 3.0–3.1 calls with neighbors and 3.1–4.3 with viewpoints. |
H3: answering context grows with the size of the asset the AI can see | Supported | Answering context was larger on the broad asset in 15 of 16 paired comparisons: strongly on viewpoints (+85% to +256%), less on reports (+4% to +49%). The exception: Opus 4.6 on reports where no owner exists (−5%). |
H4: the AI narrows data on the server instead of reading whole assets | Supported | For reports: in all 4 combinations of question kind and report size, reports of about 470,000 to 5.3 million tokens are read as calls of a median 320–1,240 tokens. Viewpoint calls return their full traversal from the capability asked about, bounded by the viewpoint rather than by filtering. |
H5: the number of tool calls is a property of the task, with a fixed minimum sequence | Not supported as stated | Of 40 configurations (model × retrieval method × asset size × question kind), 9 averaged within half a call of the minimum, all Opus 5.5 or Sonnet 5.5. 26 were above it, up to 16.5 calls on reports where no owner exists, often from a lookup per application; 5 were below it: viewpoints, skipping the definition read. |
H6: accuracy depends on the AI turning the question into the correct metamodel structure; errors increase with the number of plausible but wrong structures next to the correct one, regardless of answering context size | Partly | The path has one direction, but the answer type has two forms: an application can be owned by a person or by a team. 20 of the 23 owners left out are teams, although teams are only 16 of the 358 expected owners. On questions with a team owner, correct answers are lower in 8 of 20 combinations (equal in 11); Sonnet 4.6 drops from 99% to 78%. |
H7: retrieval methods whose asset already encodes the structure (viewpoints, reports) produce fewer mapping errors than methods where the AI builds the path itself (neighbors) | Partly | Neighbors is lowest only for Sonnet 4.6 (94%, the one combination measurably below the best). For the other three models every method scores 98–100% where owners exist and the differences are within the 95% ranges. |
5. What this means for you
Prefer a recent model for multi-step questions. Opus 5.5 and Sonnet 5.5 answered every "no owner" question correctly and used close to the minimum number of calls.
Curate viewpoints for questions that span two relationships. Curated viewpoints gave the same accuracy with a fraction of the context.
With older models, check list answers for completeness. The most common error is a missing owner, not an invented one.
6. Limits of these results
Two-hop questions only (capability → applications → owners). Single-hop, three-hop and other question types are reported separately.
Each question was asked once, so consistency is not yet measured.
A synthetic test organization; results on a real customer's data may differ.
Models at their default effort and thinking settings.
The Ardoq Assistant only; no MCP runs in this benchmark.
32 questions lost to test-infrastructure errors are left out (tool exhaustion or timeouts), and 59 Sonnet 5.5 answers were scored from traces after the evaluation endpoint rejected them. The token counts of those 59 answers were not recorded.
Report call sizes are estimated from sample requests and cover 1,482 of about 1,600 report questions. Viewpoint sizes cover only the benchmark viewpoint each answer came from, which leaves about two thirds of viewpoint questions (1,023 of about 1,600). Most viewpoint sizes are the size recorded during the run, corrected upward: on 80 calls also measured directly in Ardoq, the recorded size was 7–10% lower than the measured one, so recorded sizes were raised by 8–11%. This affects only the viewpoint call sizes shown on this page; it does not change answer accuracy, answering context, tokens per question or the conclusions.
In the no-owner questions none of the applications realizing the capability has an owner, so the correct answer is an empty list. Questions where only some of the applications lack an owner were not tested.












