Skip to main content

Three-hop retrieval benchmark results 2026-10-24

Results of running three-hop ("who are the owners of the applications integrated with this application") retrieval benchmarks

Written by Jason Baragry

We asked Ardoq AI 200 three-hop questions of the form "Who are the owners of the applications that are integrated with the 'application' application?". To answer, the AI has to follow the named application to its interfaces, then to the applications on the other side of those interfaces, then to their owners. In 100 questions the integrated applications have owners; in 100 they have none. The questions went to the Ardoq Assistant with Claude Opus 4.6, Opus 5.5 and Sonnet 5.5, between October 21 and 24, 2026. Of 3,000 questions asked, 2,987 are scored; 13 that failed because of a test-infrastructure issue (tool exhaustion or timeout) are left out. Measured in references in the Ardoq graph, the answer is three hops away: from the application to an interface, from the interface to the integrated application, and from that application to its owners.

Goals, method and research questions are on Retrieval Benchmarks.

Summary of results

Quality

  • Opus 5.5 answers almost every question correctly: 99–100% where owners exist (100% in 4 of 5 combinations) and 95–100% where none do, in every combination of retrieval method and asset size.

  • Sonnet 5.5 is reliable on reports and viewpoints but weaker with neighbors. Where owners exist it scores 98–100% on reports and viewpoints but 84% with neighbors; where no owner exists it scores 86–99%.

  • Opus 4.6 struggles with this question type. With neighbors it answered only 24% correctly where owners exist, leaving owners out in 73 of 99 answers. Where no owner exists, it named people who are not owners of the integrated applications in 159 of 496 answers (usually real owners of other applications nearby): from 9% to 45% depending on the retrieval method and asset size.

  • These differences are real, not chance. A combination counts as measurably worse when the top of its 95% range is below the bottom of the best combination's range. By that test, 5 of 15 combinations where owners exist, and 7 of 15 where none do, are measurably worse; all are Opus 4.6 (9 of its 10 combinations) or Sonnet 5.5.

  • Most wrong answers come from incorrectly inferring the metamodel structure from the question. The most common cause is listing the owners of other applications that connect to the same interface, rather than of the integrated applications: 40–44% of the extra names. Where no owner exists, questions with such applications nearby are answered correctly far less often (Opus 4.6 30% against 86%). See “Likely causes of wrong answers” below.

  • 10 questions did not finish: 6 with Sonnet 5.5 (time limit, broad reports, no owner) and 4 with Opus 4.6 (2 time limit, 1 repeated the same calls until the step limit, 1 malformed request). Each question was asked once, so consistency is not measured yet.

How to read it. Each bar is one model, with every retrieval method and asset size combined: 500 questions per bar. Green is the share answered correctly, red the share answered wrongly, and purple the share that did not finish (the AI ran out of steps or time before answering). The left panel covers questions where owners exist, the right panel questions where none do.

What it shows. Opus 5.5 is correct on almost every question. Opus 4.6 gets about a fifth wrong where owners exist and a third wrong where none do.

How to read it. Each row is one combination of model, retrieval method and asset size. The dot is the share of about 100 questions answered correctly; questions that did not finish are left out of this share and counted in the next chart. The line is the 95% range: because we tested 100 questions rather than every possible one, the true rate could plausibly lie anywhere along the line. Even 100 out of 100 correct gives a range of 96–100%. Where two lines overlap, the difference could be chance; where they don't, it is real.

What it shows. Opus 5.5's lines sit near 100% throughout. Opus 4.6's lines are far from them, especially with neighbors where owners exist and with reports where none do.

How to read it. Each bar counts the failed questions, out of 100, for every combination with at least one failure, for both question kinds. Red is a wrong answer; purple is a question that did not finish because the AI reached its step limit or time limit.

What it shows. Almost all failures are wrong answers rather than unfinished questions, and most come from Opus 4.6.

How to read it. Three-hop answers are lists of owners, so an answer can be partly right. For questions where owners exist, the left panel counts answers that left out at least one expected owner; the right panel counts answers that named someone who is not an owner. One answer can do both, so the two panels do not add up to the number of wrong answers.

What it shows. Opus 4.6 with neighbors leaves owners out on most questions. Naming someone who is not an owner is almost only Opus 4.6: Opus 5.5 never did it, and Sonnet 5.5 did it in 3 answers with neighbors.

Latency

  • A typical question takes 18–35 seconds where owners exist, and 15–88 seconds where none do (median per combination).

  • Viewpoints are quickest and reports slowest: about 21–24 seconds against 37–47 seconds, averaged over asset sizes and question kinds.

  • Slow questions are common on reports. 352 of 2,977 answered questions took more than 60 seconds, 330 of them on reports.

  • Opus 5.5 stays at the minimum number of calls on neighbors and viewpoints; Opus 4.6 makes extra calls almost everywhere, up to 27 retrieval calls per report question where no owner exists.

How to read it. Each bar is a typical time per question for one model and retrieval method: the median, averaged over asset sizes and both question kinds.

What it shows. Viewpoints are the quickest method for every model; reports are the slowest.

How to read it. Each bar is the median time per question: half the questions were faster, half slower. The black tick is the 90th percentile: 9 in 10 questions were faster than this, so the gap between bar and tick shows how long the slow ones take. Colors are models; rows are retrieval methods and asset sizes. The top panel is questions where owners exist, the bottom panel questions where none do.

What it shows. Report questions with no owner are by far the slowest, at 50–90 seconds typical for Opus models.

How to read it. A tool call is one request the AI makes to Ardoq. A retrieval call is one that fetches or finds data; metamodel lookups and link-building calls are not counted. Each bar is the average number of retrieval calls per question; the black tick is the minimum a question needs: 4 for every method (neighbors: find the application, read its neighbors, its interfaces' neighbors, and the integrated applications' neighbors; reports and viewpoints: find the application, list the assets, read a definition, get the data). Bars past the tick are extra calls; bars short of it mean a step was skipped. Same layout as the time chart.

What it shows. Opus 5.5 and Sonnet 5.5 sit at or near the minimum on neighbors and viewpoints. Reports take extra calls for every model, and many more on questions with no owner.

Cost indication (tokens)

  • Ardoq AI reads only what it needs from reports. The reports hold 6 to 73 million tokens; each report call returned a median of 360–870 tokens. Viewpoint calls return the traversal from the application asked about: 2,600–15,000 tokens.

  • Answering context is 17,000–44,000 tokens (median per combination). The largest single answering context was 449,000 tokens.

  • Tokens per question range from about 53,000 to 479,000 (mean per combination). Average by model: Opus 5.5 122,000; Opus 4.6 208,000; Sonnet 5.5 235,000.

How to read it. For report questions: blue is the size of the whole report, green what the model held when it answered, both as the median per question. The scale is logarithmic: each gridline is ten times the one before, so a bar one gridline longer is ten times larger.

What it shows. A broad report of about 73 million tokens and a curated one of 6.2 million (questions with owners) to 7.2 million (questions without) are both answered from 24,000–28,000 tokens. The bars show the median across both question kinds.

How to read it. A token is roughly three-quarters of a word: the unit AI models read and are billed in. Answering context is how much the model was holding (the question, its instructions and everything retrieved so far) when it answered. Bars are the median per question, in thousands of tokens, with both question kinds combined.

What it shows. Every combination answers from 17,000–44,000 tokens; neighbors and broad viewpoints need the most.

How to read it. Blue is what the AI could have read: the whole report, or for viewpoints what the viewpoint calls returned (a viewpoint runs from the application the AI passes, so there is no larger whole to read). Orange is what the last report or viewpoint call returned; green is what the model held when it answered. Log scale, as in the token chart above.

What it shows. Report calls return a few hundred tokens of reports holding millions. Viewpoint calls return their whole traversal, a few thousand tokens.

How to read it. Each bar is the total tokens per question, summed over every request the AI made. Each tool call adds to the conversation the model re-reads, so this grows with the number of calls. Same layout as the time chart. The Assistant reuses cached prompt tokens, so its counts overstate what it is billed.

What it shows. Neighbors where owners exist and reports where none do are the most token-hungry. Opus 5.5 uses the fewest tokens in most combinations.

What we tested

  • Questions: "Who are the owners of the applications that are integrated with the 'application' application?" Two applications are integrated when an interface connects them, in either direction. 100 applications whose integrated applications have owners (the expected answer is the full set of owners, often several) and 100 whose integrated applications have none (the expected answer is no owner). Expected answers come from the data itself, not written by hand.

  • Access: the Ardoq Assistant only. This benchmark has no MCP runs.

  • Retrieval methods: neighbors (walk the graph from the application), reports, and viewpoints.

  • Asset size: each report and viewpoint question is asked against a curated asset (narrow and sparse) and a broad, dense one, with the same question and expected answer.

  • Models: Claude Opus 4.6, Opus 5.5 and Sonnet 5.5, at the Assistant's default settings. Sonnet 4.6 was also started, but its runs were stopped because it repeatedly looped until it reached the 100-call limit, which also overloaded the test service; its results are not included.

  • Scoring: strict. The names returned must match the expected names exactly; one missing or one extra name fails the question. A partial-credit score (F1) is also reported, because three-hop answers are lists. Correct-answer rates carry 95% ranges.

  • Repeats: each question was asked once per configuration.

  • Corrections after tracing failed questions: 13 questions that failed because of a test-infrastructure error are left out of every rate. 61 Sonnet 5.5 answers that the evaluation endpoint rejected only because of their format were scored from the answer recorded in the trace: 58 called a tool that does not exist before answering, and 3 answered without the list of names. All 61 were correct. 3 further Sonnet 5.5 answers of the first kind were wrong and are scored as wrong answers. Questions that looped or timed out stay in as "did not finish".

Discussion by research question

RQ1: How reliable is Ardoq AI at retrieval? This covers accuracy (how often the answer is right) and consistency (whether the same question gets the same answer when asked again), on questions that have an answer and on questions that do not.

  • It depends strongly on the model. Opus 5.5 is reliable on this question type; Sonnet 5.5 is reliable except with neighbors and on some questions with no owner; Opus 4.6 is not reliable.

  • This is the hardest question type so far: every model that also ran the two-hop benchmark scores lower here, with the answer one hop further away.

  • Opus 4.6 fails in both directions: it leaves owners out where they exist, and names owners where none exist.

  • Consistency is not measured: each question was asked once.

Likely causes of wrong answers

The correct answer combines two paths: applications that connect to one of the named application’s own interfaces, and the parent applications of the interfaces the named application connects to. Every name in a wrong answer was compared with the graph around the question, to see which structure it belongs to.

  • Owners of other applications on the same interface (connected to the interface, not its parent): 40% of the extra names where no owner exists and 44% where owners exist. This is the clearest case of incorrectly inferring the metamodel structure from the question. Where no owner exists, 32 of 100 questions have such applications nearby, and correct answers drop on them: Opus 4.6 86% → 30%, Sonnet 5.5 97% → 87%, Opus 5.5 99.7% → 94%. The drop holds at the same answering context, so it is not a context-size effect. Where owners exist, these applications are always nearby and do not lower accuracy.

  • The named application’s own owners: 26% of the extra names where owners exist, almost all from Opus 4.6, and 2% where no owner exists.

  • Applications with the same name as an interface on the path: almost every interface not named “[INT] … API” shares its name with an application, but its owners make up only 4–6% of the extra names.

  • Experts are not a cause: only 1 extra name is a person who is an expert in an application nearby.

  • Missing owners: 213 expected owners were left out, 173 of them by Opus 4.6 with neighbors. They are spread evenly over the two paths (108 and 105), so the AI did not simply follow one direction only.

  • Long lists on report questions: a few answers list many names that belong to none of these structures; 10 answers hold 56% of them, mostly Opus 4.6 on reports where no owner exists. Reading the wrong rows of a report is a likely cause, but the traces have not been checked.

  • A duplicated application name (Forte Analytics (SaaS)) caused 9 answers to combine two applications; those answers are accepted (see Limits).

Together, the measured structures fully explain 56% of the wrong answers where no owner exists and 66% where owners exist. The newer models largely avoid these mistakes: Opus 5.5 named someone from a wrong structure in only a few no-owner answers.

In general graph terms. These causes are not specific to ownership or integrations. The question defines a path pattern (start node, typed edges, directions, answer type), and the errors come from near misses: paths that reach nodes of the answer type but differ slightly from that pattern. In this benchmark:

  • One edge flipped at a shared node (applications on the same interface): same edge types and depth as the correct path, one direction different. This was by far the largest source of wrong names.

  • A truncated path (the named application’s own owners): physically closer, but a clearly different pattern, and mostly avoided by the newer models.

  • Identity confusion (an interface named like an application) and an ambiguous starting node (a duplicated name): rarer.

  • Edge-type substitution (experts instead of owners): not seen here.

The single-hop and two-hop questions show two further kinds: a starting name that nearly matches another component (an application name with and without “(v2)”), and answer-type narrowing (teams left out as owners).

What made a question hard was structural, not size: how similar the near miss is to the correct path, how many answer-type nodes a shared high-degree node exposes, and above all whether the correct path returns nothing while a near miss returns something. Questions where owners exist had heavy near-miss exposure but no loss of accuracy; the correct path gave the AI something to anchor on. The direction changes in the integration pattern add difficulty that single-hop and two-hop questions do not have. Retrieval methods change how much of this the AI must resolve itself: neighbors exposes every near miss at every step, while viewpoints and reports encode the pattern.

Expected in other scenarios (not yet tested): any multi-hop question with a shared intermediate node and a same-depth near miss should show the same errors, for example dependencies through a shared database, processes linked through a shared capability, applications on a shared server, or parent and child units in an organization. Errors should concentrate on questions where the correct answer is empty, and a per-question near-miss exposure measure should predict them better than answering context.

RQ2: Does the size of the Ardoq asset the AI retrieves (viewpoint, report, etc.), and therefore the context size, affect retrieval accuracy?

  • For Opus 4.6, yes on viewpoints; for the newer models, no.

  • Opus 4.6 on broad viewpoints answered 69% of no-owner questions correctly against 91% on curated viewpoints (−22 points), and 6 points fewer where owners exist.

  • Opus 5.5 and Sonnet 5.5 change by at most 2 points where owners exist (Sonnet 5.5 on broad reports: −2), and are unchanged or better on broad assets where no owner exists (up to +10 points on Sonnet 5.5’s no-owner report questions).

  • Answering context grows on broad assets in most combinations, from +14% to +99%, most on viewpoints with no owner. Time per question mostly rises too.

How to read it. Each dot compares the same questions on a curated asset and a broad one, for one combination of model, method and question kind. Left: change in correct answers, in percentage points. Middle: change in answering context, in percent. Right: change in time, in percent. The vertical line is no change; dots to the right mean the broad asset scored higher, used more tokens or took longer.

What it shows. Opus 4.6's no-owner viewpoint questions fall furthest to the left. Context moves to the right almost everywhere, most for viewpoints.

RQ3: Is it beneficial to curate assets to get more reliable retrieval? A curated asset is narrow and sparse: its information content is specific to the retrieval question.

  • Yes for older models, and for context with every model. Curated viewpoints kept Opus 4.6 at 91% on no-owner questions where broad viewpoints dropped it to 69%.

  • For Opus 5.5 and Sonnet 5.5, curated and broad assets give the same accuracy, but curated viewpoints need less context: broad viewpoints raise it by 61–99% on no-owner questions.

RQ4: Is answer quality different between Assistant and MCP access to the same tools?

Not tested in this benchmark: only the Ardoq Assistant was run. The single-hop results compare the Assistant and MCP.

Latency compared with the expected minimum

  • Neighbors: Opus 5.5 and Sonnet 5.5 use about the minimum of 4 retrieval calls (fewer on no-owner questions, where the walk ends early). Opus 4.6 uses 5–7, adding extra name lookups.

  • Reports: above the minimum for every model. Where owners exist, Opus 5.5 uses 4.8–5.2 calls, Sonnet 5.5 7.2–9.2 and Opus 4.6 9.8–11.4, looking up integrated applications by name one at a time. Where no owner exists, calls rise to 9–18 for Opus 5.5, 11–17 for Sonnet 5.5 and about 27 for Opus 4.6, as the models search more assets before concluding.

  • Viewpoints: Opus 5.5 often skips the definition read (3.6–3.9 calls, below the minimum) and still answers correctly; Sonnet 5.5 sits at about 4; Opus 4.6 uses 6–9.

  • Opus 5.5 against Opus 4.6: half the calls on no-owner questions and 21% less time; where owners exist, 20% fewer calls but 21% more time per question (median).

Token cost in more detail

  • Tokens per question follow the number of calls and the size of what each call returns: neighbors has to walk every interface of the application, and no-owner report questions make the most calls.

  • Opus 5.5 uses 14% fewer tokens than Opus 4.6 where owners exist and 50% fewer where none do (median per combination).

  • The Assistant reuses cached prompt tokens, so its billed cost is lower than its token counts suggest. Token counts are shown, not prices.

Hypotheses: what the evidence says

Hypothesis

Verdict

Why

H1: accuracy holds near ceiling up to roughly 256k tokens of answering context and falls visibly beyond roughly 500k

Not testable yet

One answering context reached 449,000 tokens, but medians are 17,000–44,000. The accuracy differences here follow the model, not the context size.

H2: neighbors needs fewer tool calls than reports or viewpoints

Supported

Stated for single-hop; checked here on three-hop, where every method needs at least 4. Comparing the same model and question kind, neighbors used fewer retrieval calls than reports in 12 of 12 comparisons and fewer than viewpoints in 10 of 12. Exception: Opus 5.5 where owners exist, which used slightly fewer with viewpoints (3.6 against 4.0).

H3: answering context grows with the size of the asset the AI can see

Supported

Answering context grew on the broad asset in 11 of 12 comparisons (+14% to +99%, most on viewpoints). Exception: Opus 5.5 on no-owner report questions (−27%).

H4: the AI narrows data on the server instead of reading whole assets

Supported

In all 4 report comparisons (two asset sizes, two question kinds), reports of 6 to 73 million tokens were read as calls of 360–870 tokens. Viewpoint calls return their traversal from the application asked about.

H5: the number of tool calls is a property of the task, with a fixed minimum sequence

Not supported as stated

Retrieval calls were within half a call of the minimum in only 10 of 30 combinations (5 each for Opus 5.5 and Sonnet 5.5) and above it in all 10 of Opus 4.6's, so the count depends on the model. Viewpoints also undercut the minimum by skipping the definition read.

H6: accuracy depends on the AI turning the question into the correct metamodel structure; errors increase with the number of plausible but wrong structures next to the correct one, regardless of answering context size

Partly

Where no owner exists, 32 of 100 questions have other applications connected to the same interface as the correct path. On those questions correct answers are lower in 12 of 15 combinations (equal at 100% in the other 3), at the same answering context: 72% against 92% at 25,000–40,000 tokens. Where owners exist, every question has such applications nearby (13 to 394 wrong owners within reach), and more of them did not lower accuracy (lower in 3 of 15 combinations). The measured wrong structures fully explain 56% of wrong answers where no owner exists and 66% where owners exist.

H7: retrieval methods whose asset already encodes the structure (viewpoints, reports) produce fewer mapping errors than methods where the AI builds the path itself (neighbous)

Supported

Neighbors, where the AI builds the path itself, has the lowest correct-answer rate for all three models: Opus 4.6 45% against 74–94%, Sonnet 5.5 87% against 93–99%, Opus 5.5 97% (tied with curated reports) against 97–100%. Viewpoints and reports reduce the errors but do not remove them: answers on both also named owners from wrong structures.

5. What this means for you

  • Use a recent model for questions that follow integrations. Opus 5.5 answered these correctly where Opus 4.6 often left owners out or invented them.

  • Prefer viewpoints for sophisticated questions: they were the quickest method for every model and, over both question kinds, at least as accurate as reports and neighbors for every model.

  • With older models, avoid neighbors for two-step questions, and curate viewpoints: curated viewpoints kept Opus 4.6 far more accurate than broad ones.

6. Limits of these results

  • Each question was asked once, so consistency is not yet measured.

  • A synthetic test organization; results on a real customer's data may differ.

  • Models at their default effort and thinking settings; Sonnet 4.6 could not complete its runs and is not included.

  • The Ardoq Assistant only; no MCP runs in this benchmark.

  • 13 questions lost to test-infrastructure errors are left out (tool exhaustion or timeouts); 61 Sonnet 5.5 answers rejected only for their format were scored from traces, and 3 were rescored as wrong after tracing. The token counts of those 61 answers were not recorded.

  • Duplicate application name: two Applications in the test data are named “Forte Analytics (SaaS)”, and one question names it. The answer key covers one of them (5 integrated applications, 5 owners); the other has 9 integrated applications whose 9 owners are different people. In 9 of the question’s 15 answers (four Sonnet 5.5, three Opus 5.5, two Opus 4.6) the AI found both and listed the owners of both. Because the question cannot tell the two apart, those answers are accepted as correct; they were first scored as wrong. This is the only question in the three benchmarks that names a duplicated application. Separately, one Sonnet 5.5 answer on another question added six extra names.

  • Report call sizes are estimated from sample requests and cover only the questions whose last report request could be sized: 83 of about 600 questions on the broad report and 356 of about 600 on the curated one. Viewpoint call sizes cover 918 of about 1,200 viewpoint questions. Most are the size recorded during the run, corrected upward: on a sample of calls also measured directly in Ardoq, the recorded size was 7–9% lower than the measured one, so recorded sizes were raised by 8–10%. This affects only the viewpoint call sizes shown on this page; it does not change answer accuracy, answering context, tokens per question or the conclusions. Opus 4.6 questions where no owner exists are left out of the viewpoint sizes because their recorded sizes did not match.

  • In the no-owner questions none of the integrated applications has an owner, so the correct answer is an empty list. Questions where only some integrated applications lack an owner were not tested.

  • Causes of wrong answers are matched by name against the graph around each question (applications on the same interface, the named application’s own owners, same-name applications and experts). Names that match none of these, mostly long lists on report questions, have not been traced.

Did this answer your question?