What this covers. 6,000 questions run between September 23 and October 7, 2026: single-hop questions ("who owns application X?"), answered by Ardoq AI through the Ardoq Assistant and through an agent connected to the Ardoq MCP server, with Claude Opus 4.6, Sonnet 4.6 and Opus 5.5. Goals, method and research questions are on Retrieval Benchmarks.
Summary of results
Quality
When an answer exists, Ardoq AI finds it. Every one of the 30 configurations (access route × model × retrieval method × asset size) answered 99–100% of these questions correctly. Across all 3,000 such questions, 2 answers were wrong, each one owner short.
When no answer exists, the Assistant says so. The Assistant answered "no owner" correctly in all 15 of its configurations (100%).
An MCP-connected agent sometimes names someone who is not an owner, usually the owner of a near-identically named application. This is concentrated in Sonnet 4.6: 82% correct on curated reports, 91% on broad reports and broad viewpoints. Opus 5.5 over MCP got 1 wrong out of 500.
Two kinds of failure. 44 answers named someone who is not an owner (all MCP). 35 questions did not finish: 33 MCP runs hit the agent's step limit while still searching (31 of them Sonnet 4.6 on viewpoints), and 2 Assistant runs hit a server error.
How to read it. Each row is one combination of model, retrieval method and asset size; blue dots are the Assistant, orange dots MCP. The left panel covers questions where an owner exists, the right panel questions where none does. The dot is the share of about 100 questions answered correctly. The line is the 95% range: because we tested 100 questions rather than every possible one, the true rate could plausibly lie anywhere along the line. Even 100 out of 100 correct gives a range of 96–100%. Where two lines overlap, the difference could be chance; where they don't, it is real.
What it shows. On the left, every line overlaps near 100%, so all combinations perform alike. On the right, MCP with Sonnet 4.6 falls to 82–91%, and its lines no longer overlap the Assistant's.
How to read it. Each bar counts the failed questions, out of 100 with no answer, for every combination with at least one failure. Red is a wrong answer, usually the owner of a near-identically named application. Purple is a question that did not finish: the AI ran out of steps while still searching, or the server returned an error.
What it shows. Failures are concentrated in MCP with Sonnet 4.6. The Assistant had two, both server errors.
Latency
Typical time per question (median): 7–15 s with neighbors, 13–35 s with reports, 7–29 s with viewpoints.
Questions with no answer take longer and need more tool calls, because the AI has to rule possibilities out before it concludes there is nothing to find.
Slow outliers are rare. 66 of 6,000 questions took over 60 s (13 Assistant, 53 MCP), mostly no-answer questions with Sonnet 4.6.
Opus 5.5 is faster than Opus 4.6 on the same questions: a median 17% less time per configuration, and up to 34% (Assistant) or 51% (MCP) less. It also comes closest to the minimum number of tool calls.
How to read it. Each bar is the median time per question: half the questions were faster, half slower. The black tick is the 90th percentile: 9 in 10 questions were faster than this, so the gap between bar and tick shows how long the slow ones take. Colors are models; rows are retrieval methods and asset sizes. The left panels are the Assistant, the right panels MCP; the top row is questions with an answer, the bottom row questions without. The left and right columns use different scales.
What it shows. In the Assistant, neighbors is quickest; over MCP, viewpoints are about as quick. Questions with no answer take longer, and their slowest questions much longer, especially with Sonnet 4.6 over MCP.
How to read it. A tool call is one request the AI makes to Ardoq, such as finding an application or fetching report data. Each bar is the average number per question; the black tick is the minimum a question needs (2 for neighbors, 4 for reports and viewpoints). Bars past the tick are extra calls; bars short of it mean a step was skipped. Every tool call is counted, including metamodel lookups, because this run’s data does not separate retrieval calls. Same layout as the time chart.
What it shows. Opus 5.5 stays closest to the minimum. Over MCP, viewpoints use fewer calls than the minimum, because the AI skips reading the viewpoint definition. Sonnet 4.6 over MCP makes many extra calls on questions with no answer.
Cost indication (tokens)
Ardoq AI reads only what it needs. The reports used here hold 1.7M–12.8M tokens of data, far more than any model can read. The AI asks Ardoq for the matching rows only, and answers holding a median of 10k–44k tokens of context.
Tokens per question (all model requests in a question) range from about 25k to 300k, depending on access route, retrieval method and model.
Opus 5.5 uses fewer tokens than Opus 4.6 in the Assistant: 34–68% fewer per question (median 45%).
How to read it. A token is roughly three-quarters of a word: the unit AI models read and are billed in. Answering context is how much the model was holding (the question, its instructions and everything retrieved so far) when it answered. Bars are the median per question, in thousands of tokens, with both question kinds combined. The Assistant and MCP panels have different scales and should not be compared with each other.
What it shows. Every combination answers from 10,000–44,000 tokens, a small fraction of what these models can hold.
How to read it. For report questions: blue is the whole report, orange what the AI's last report request returned, green what the model held when it answered. The scale is logarithmic: each gridline is ten times the one before, so a bar one gridline longer is ten times larger.
What it shows. Reports of 1.7–12.8 million tokens are answered from tens of thousands, because the AI asks Ardoq for the matching rows only.
How to read it. Each bar is the total tokens per question, summed over every request the AI made. Each tool call adds to the conversation the model re-reads, so this grows with the number of calls. Same layout as the time chart. The Assistant reuses cached tokens, so its counts overstate what it is billed.
What it shows. Tokens follow tool calls. Opus 5.5 uses the fewest in the Assistant, and MCP with Sonnet 4.6 the most on questions with no answer.
What we tested
Questions: "Who is the owner of the X application?" 100 questions where an owner exists, and 100 where the application has no owner and the correct answer is "no owner". Expected answers come from the data itself, not written by hand.
Access: the Ardoq Assistant, and an AI agent connected through the Ardoq MCP server. The MCP agent is our own reference agent; other MCP clients may behave differently.
Retrieval methods: neighbors (walk the graph from the named application), reports, and viewpoints.
Asset size: each report and viewpoint question is asked against a curated asset (narrow and sparse: little beyond what the question needs) and a broad, dense one. The question and expected answer are identical.
Models: Claude Opus 4.6, Sonnet 4.6 and Opus 5.5, at the Assistant's default settings.
Scoring: strict. The names returned must match the expected names exactly; one missing or one extra name fails the question. Correct-answer rates carry 95% ranges, and overlapping ranges are read as "no measurable difference".
Repeats: each question was asked once per configuration.
Discussion by research question
RQ1: How reliable is Ardoq AI at retrieval?
Accuracy is at the ceiling when an answer exists. 99–100% everywhere, with overlapping ranges, so these questions cannot tell configurations apart.
"No answer" questions are the real test. They separate configurations: three of 30 are measurably below the best, all MCP with Sonnet 4.6. Measurably below means the top of a configuration’s 95% range is below the bottom of the best configuration’s range for the same question kind. The failure is a confident, plausible but wrong name, which a user has no reason to doubt.
Consistency is not measured yet. Each question was asked once, so whether the same question gets the same answer when asked again remains open.
Verification in progress. 8 no-answer questions received the same wrong name from 2–3 configurations. For 2 of them we checked the data and the expected answer ("no owner") is correct; the other 6 are still being verified.
Likely causes of wrong answers
All 44 wrong answers were on questions with no answer, over MCP (38 Sonnet 4.6, 5 Opus 4.6, 1 Opus 5.5), and each named one person. Every name was compared with the graph around the question.
The owner of a near-identically named application: 28 of the 44 names. The test data has many name twins, the same application name with and without “(v2)”, and the AI answered with the owner of the twin. For example, Sofia Fernandez owns Accord Reporting (ServiceNow GRC) (v2) and was given as the owner of Accord Reporting (ServiceNow GRC). Where the twin has an owner, MCP answers are correct less often (lower in 6 of 15 combinations, equal at 100% in 8).
People linked to the application in another way: 10 names – experts (4), another linked person (1) and owners of integrated applications (5).
Other: 3 names of the owner of an unrelated application, 2 answers that gave “unowned” as a name, and 1 name not found.
In general graph terms, this is an ambiguous starting node: the name in the question is close to the name of another component of the same type, and the AI follows the wrong one. It is identity confusion, not a wrong path. Experts were only a minor source. Twin data covers 39 of the 100 questions with no answer, so the effect of twins is measured on those.
RQ2: Does asset size affect retrieval accuracy?
No change where an answer exists. Comparing the same questions on curated and broad assets, the correct-answer rate did not change for any model or access route (two rows moved by one question).
Broad assets cost more. Answering context grew by 3% to 111%, and time per question typically by 5–30%.
Why accuracy holds: the AI filters in Ardoq before reading, so a much larger asset reaches the model as only moderately more context.
No-answer questions are mixed, and only for MCP with Sonnet 4.6: reports improved by 9 points on broad assets, viewpoints dropped by 11 points, but that comparison also lost 14–17 questions per side that did not finish.
How to read it. Each dot compares the same questions on a curated asset and a broad one, for one combination. Left: change in correct answers, in percentage points. Middle: change in answering context, in percent. Right: change in time, in percent. The vertical line is no change; dots to the right mean the broad asset scored higher, used more tokens or took longer.
What it shows. Where an answer exists, correct answers stay on the line. Context, and mostly time, move to the right.
RQ3: Is it beneficial to curate assets?
Yes, for cost and speed. Curated assets gave the same accuracy with less context and usually less time.
No measurable accuracy gain on these questions, because accuracy is already at the ceiling. Harder question types may show one.
RQ4: Is quality different between Assistant and MCP access?
Equal when an answer exists.
Better for the Assistant when no answer exists. On all 30 matched comparisons (same question, model, retrieval method and asset size) the Assistant was equal or better, and it was never worse on a single question. The gap is concentrated in Sonnet 4.6 (up to 18 points).
Opus 5.5 nearly closes the gap: 1 difference in 1,000 matched questions.
What the comparison means: the two are complete systems that differ in system prompt, available tools and agent loop, so this shows what each delivers, not the effect of the access route alone.
Latency compared with the expected minimum
Each retrieval method has a minimum sequence of tool calls for a single-hop question: neighbors needs 2 (identify the application, read its neighbors), reports and viewpoints need 4 (identify the application, list the assets, read a definition to confirm it can answer, get the data). Calls beyond the minimum cost time and tokens.
Neighbors runs at the minimum of 2 calls when an answer exists, except Opus 4.6 in the Assistant, which makes one extra call on every question.
Viewpoints over MCP run below the expected minimum: about 3 calls. The AI goes straight from the list of viewpoints to the data, skipping the definition check, and still answers correctly. The expected sequence assumes a confirmation step that models often treat as optional.
Viewpoints in the Assistant run at about 4 calls with Opus 5.5 and Sonnet 4.6, and 5.7–6.5 with Opus 4.6, which adds metamodel lookups.
Reports run above the minimum everywhere: 4.7–6.6 calls in the Assistant and 6.8–7.5 over MCP. Over MCP the AI requests report data 2.5–3 times per question rather than once. Whether that is re-filtering or splitting large responses is not yet classified.
No-answer questions need more calls everywhere. Opus 5.5 stays close to the minimum (about 1 extra call on average); Sonnet 4.6 over MCP averages 7.5 extra calls, and on viewpoints keeps searching until it hits the step limit.
Time follows calls. The configurations with the most calls are the slowest, and Opus 5.5's fewer calls are a large part of its speed advantage.
Token cost in more detail
The model never reads the whole asset. Over MCP, 63–72% of report data requests named the exact application; the Assistant selected only the columns it needed in 61–95% of report requests.
Context at answer time stays far below model limits. The medians are 10k–44k tokens; the largest single answering context in all 6,000 questions was 311k.
Tokens per question are dominated by the number of calls, because each call adds to the conversation the model re-reads. Fewer calls (Opus 5.5) means fewer tokens, even where its answering context is larger.
The Assistant reuses cached prompt tokens, so its billed cost is lower than its token counts suggest. Token counts are shown, not prices.
Hypotheses: what the evidence says
Hypothesis | Verdict | Why |
H1: accuracy holds up to ~256k tokens of answering context and falls beyond ~500k | Not testable yet | No configuration came near the predicted range: the median answering context was 10k–44k in all 30 configurations, and the largest single answering context was 311k. Questions that force the AI to read many entities are needed. |
H2: neighbors needs fewer tool calls than reports or viewpoints | Supported | Neighbors used fewer tool calls in 46 of 48 comparisons (same access route, model and question kind, against each report and viewpoint asset). The exceptions: Opus 5.5 over MCP on questions with no answer, where viewpoints used 3.2–3.3 calls against 3.4 for neighbors. |
H3: answering context grows with asset size | Supported | Answering context was larger on the broad asset in all 24 paired comparisons (+3% to +111%), but grew far less than the assets do: broad assets are 5–6 times the size of curated ones. |
H4: the AI narrows data on the server instead of reading whole assets | Supported | In all 8 combinations of access route, question kind and report size, the last report request returned a median of 219–3,924 tokens from reports of 1.7–12.8 million tokens. |
H5: tool calls are a property of the task, with a fixed minimum sequence | Not supported as stated | Of 60 configurations (access route × model × retrieval method × asset size × question kind), 13 averaged within half a call of the minimum, 7 of them Opus 5.5. 40 were above it, Sonnet 4.6 far above on questions with no answer, and 7 below it: viewpoints over MCP, skipping the definition read. These counts include every tool call, including metamodel lookups. |
H6: accuracy depends on the AI turning the question into the correct metamodel structure; errors increase with the number of plausible but wrong structures next to the correct one, regardless of answering context size | Partly | There is only one relationship to follow, but the starting application can be confused with a near-identically named one: 28 of the 44 wrong names belong to the owner of the same application name with or without “(v2)”. Over MCP, questions whose name twin has an owner are answered correctly less often in 6 of 15 combinations (equal at 100% in 8; Sonnet 4.6 on curated reports 41% against 98–100%). The Assistant answered all of them correctly. |
H7: retrieval methods whose asset already encodes the structure (viewpoints, reports) produce fewer mapping errors than methods where the AI builds the path itself (neighbors) | Not supported as stated | On single-hop questions neighbous was as accurate as or more accurate than reports and viewpoints. The three combinations measurably below the best are MCP with Sonnet 4.6 on reports and viewpoints, on questions with no answer. |
What this means for you
Curate assets for recurring questions. Focused reports and viewpoints needed less context in all 24 paired comparisons and less time in 18 of them, with the same accuracy where an answer exists. The exceptions on time are mostly questions with no answer over MCP.
Use neighbors for single lookups. In the Assistant it was the fastest and used the fewest tokens for every model and question kind. Over MCP, viewpoints were as fast and as lean for Opus 5.5 on questions with no answer, and for Opus 4.6 on some viewpoint sizes.
If you connect your own agent through MCP, prefer a newer model such as Opus 5.5, and treat an answer that names an owner for something that may have none with care.
Limits of these results
Single-hop questions only. Two-hop, multi-hop and aggregation questions come next.
One run per question, so consistency is not yet measured.
A synthetic test organization; results on a real customer model may differ.
Models at default effort and thinking settings only.
MCP results come from our reference agent; other MCP clients may behave differently.
6 of the expected "no owner" answers that several configurations disagreed with are still being verified.








