Purpose
Expectations of what AI can reliably do when connected to an enterprise architecture repository are overinflated. Industry benchmarks (such as MRCR v2, GraphWalk, and AA-Omniscience Hallucination) show that retrieval-augmented AI systems hallucinate, retrieve inconsistently, and often fail to recognize when they lack the information to answer. Accuracy also degrades as context grows.
This benchmark investigates whether the same limitations appear when Ardoq AI retrieves entity data from Ardoq, and how far limitations can be reduced by changing how the AI accesses that data.
Retrieval matters beyond standalone questions: it is often the foundation step that later reasoning depends on. E.g., an agent helping with disaster recovery planning must first reliably identify the applications and infrastructure connected to a given application, and the owners of those components, before it can reason about impact, priority or recovery order. If retrieval is unreliable, everything built on top of it inherits that unreliability, however good the reasoning.
Scope:
Retrieval benchmarks encompass graph-based retrieval of information (single-hop, multi-hop) and negative rejection over a known metamodel. Time-based queries and metamodel/ontology variants are separate benchmarks.
Results
Result pages cover separate scenarios and states which research questions it addresses.
Summary and key takeaways (2026-10)
Three scenarios were run: single-hop (Ardoq Assistant and MCP), two-hop and three-hop (Assistant). Each question was asked once, on a synthetic test organisation, with AI models at the default settings used by Ardoq. Counts and figures are on the result pages.
What we can conclude
Ardoq AI finds the data reliably. The AI reliably narrows reports and viewpoints on the server and answers from a small part of each asset, even when the asset holds millions of tokens.
Retrieval methods that encode the structure help on harder questions. Viewpoints and reports beat neighbours on three-hop questions, but not on single-hop.
Curated assets reduce context and usually time; they improve accuracy only where it is not already near the ceiling.
Accuracy depends on the model and on how ambiguous the graph is around the question. The newest model is close to fully correct in every scenario; the older models lose accuracy as the number of hops grows.
Wrong answers come from incorrectly inferring the metamodel structure from the question. Each scenario has a dominant “near-miss”: a starting name that nearly matches another application (single-hop), teams not treated as owners (two-hop), and owners of other applications that connect to the same interface (three-hop).
Questions with no answer are the harder test. When the correct structure returns nothing, models tend to report what a near-miss returns instead of saying there is no answer. When answers exist, near-misses rarely cause errors.
The effect of very large answering contexts (see Hypothesis 1) has not yet been tested: no scenario has reached the predicted range due to automatic filtering of data.
Hypothesis | Single-hop | Two-hop | Three-hop |
H1 | Not testable yet | Not testable yet | Not testable yet |
H2 | Supported | Supported | Supported |
H3 | Supported | Supported | Supported |
H4 | Supported | Supported | Supported |
H5 | Not supported as stated | Not supported as stated | Not supported as stated |
H6 | Partly | Partly | Partly |
H7 | Not supported as stated | Partly | Supported |
Likely sources of error across the three scenarios: near-identical or duplicated names, which are realistic data ambiguity (for example the same application name with and without “(v2)”); answer types left out (teams as owners); and paths through shared intermediate nodes such as an interface used by many applications.
How to read it. Each panel is one retrieval method and one question kind (an answer exists, or no answer exists). Along the bottom are the three scenarios, by number of hops; each is a separate set of questions. The dot is the share of questions answered correctly and the line is its 95% range. Colours and marker shapes are models. Ardoq Assistant only, because MCP ran on single-hop only.
What it shows. The newest model stays close to fully correct in every scenario; the older models fall behind on the three-hop questions, most with neighbours. Because each hop count is a different scenario with its own questions and data, the chart compares scenarios; it does not isolate the effect of the number of hops.
How to read it. Three-hop questions where no owner exists only. Along the bottom is how many wrong owners are reachable through an interface that the named application shares with other applications (a near-miss): none, a few, or many, with the number of questions in each group. Panels are retrieval methods; colours and marker shapes are models; lines are 95% ranges.
What it shows. Once any such near-miss exists, correct answers drop: sharply for Opus 4.6, less for the 5.5 models, and not at all for Opus 5.5 on viewpoints. The step from none to some matters more than the number of wrong owners. Questions where owners exist do not show this effect, and the questions were not chosen to vary exposure, so this is an observed association within one scenario.
What this means for Ardoq users
Use curated viewpoints and reports for recurring questions that span several relationships. An asset that encodes the path through the graph leaves the AI fewer wrong turns, and a focused asset also keeps the context small and answers fast. Walking the graph from a component is fine for simple, direct lookups.
Be careful with answers to questions that may have no answer. When nothing matches, the AI is most likely to return a plausible result from a nearby component instead of saying there is none. Check such answers against the component they came from, especially for ownership or accountability.
Watch out for duplicate and similar names. Components of the same type with similar or identical names are normal in a real repository, but the AI can confuse them. Where a name is ambiguous, say which component you mean, for example by adding its workspace or another distinguishing detail.
Use naming conventions that keep different component types apart. A component that carries the name of a component of another type invites the AI to follow the wrong one; a type-specific convention, such as a prefix for interfaces, removes that similarity.
Say exactly what you mean in the question. Name the component types and the relationship, and include every kind of answer you expect, for example “the people or teams that own …”.
Check list answers for completeness. When a list answer is wrong, it is more often missing an item than adding one.
If you connect your own agent through MCP, test it on questions that have no answer before relying on it; agents differ most on these questions.
What "Ardoq AI" means here
For the first single-hop benchmark, Ardoq AI means both ways of accessing AI queries in Ardoq:
Ardoq Assistant: the AI assistant in the Ardoq product.
Ardoq MCP: an AI agent connected to Ardoq through the Ardoq MCP server.
Both are tested on the same questions. If the first benchmark shows no significant quality difference between them, later benchmarks may test the Assistant only.
“Asset” used a collective term for viewpoints, reports, and dashboards. These have been used as the primary means of retrieving sets of information by AI.
Research questions
RQ1: How reliable is Ardoq AI at retrieval? This covers accuracy (how often the answer is right) and consistency (whether the same question gets the same answer when asked again), on questions that have an answer and on questions that do not.
RQ2: Does the size of the Ardoq asset the AI retrieves (viewpoint, report, etc), and therefore the context size, affect retrieval accuracy?
RQ3: Is it beneficial to curate assets to get more reliable retrieval? A curated asset is narrow and sparse: its information content is specific to the retrieval question.
RQ4: Is answer quality different between Assistant and MCP access to the same tools?
Hypotheses
Existing, external AI benchmarks create a starting set of assumptions for Retrieval quality and reliability analysis
ID | Hypothesis |
H1 | Accuracy holds near ceiling up to roughly 256k tokens of answering context and falls visibly beyond roughly 500k. |
H2 | On single-hop questions, the neighbours toolset needs fewer tool calls than reports or viewpoints, which must first find the right asset. |
H3 | Answering context grows with the size of the asset the AI can see. |
H4 | The AI narrows data on the server (filters, column selection, bounded traversals) instead of reading whole assets. |
H5 | The number of tool calls is a property of the task, not of the model: each toolset and question type should result in a fixed minimum sequence of tool calls. Extra tool calls at the asset-selection steps indicate inefficient inference. |
H6 | Accuracy depends on the AI turning the question into the correct metamodel structure. Errors increase with the number of plausible but wrong structures next to the correct one, regardless of answering context size. A wrong structure is a near -miss: a path that reaches nodes of the answer type but differs slightly from the path the question defines. Types of near-miss:
The risk is expected to be highest when the near-miss has the same depth as the correct path, runs through a high-degree shared node, and the correct path returns nothing while the near-miss returns nodes of the answer type. |
H7 | Retrieval methods whose asset already encodes the structure (viewpoints, reports) produce fewer mapping errors than methods where the AI builds the path itself (neighbours). |
Minimum tool sequences
For a single-hop question, each toolset has a fixed minimum sequence of tool calls:
Toolset | Minimum calls | Sequence |
neighbours | 2 | 1) identify the named component 2) get_neighbours for that component |
reports | 4 | 1) identify the named component 2) list the reports 3) get a report definition to confirm it can answer the question 4) get the report data |
viewpoints | 4 | 1) identify the named component 2) list the viewpoints 3) read a viewpoint definition to confirm it can answer the question 4) populate the viewpoint to get the data |
The goldens use component names that exist and are unique, so step 1 needs a single call. Calls beyond the minimum are classified, not just counted:
Mistake: asset selection. Extra list or definition calls (steps 2 and 3) mean the AI chose an asset that could not answer the question, or did not settle on one.
Valid outcomes, detected and reported separately:
Size-limit split: a data call (step 4) whose response is too large is split into several smaller calls.
Infrastructure retry: a call repeated after a timeout or server error.
Verification on negative questions: after finding no answer, the AI checks a further asset before concluding that none exists.
Repeated call: the same call made again with the same arguments.
Benchmark setup
What varies and what is held fixed
Varies | Values |
Question type | single-hop, negative (no valid answer), two-hop; later multi-hop and aggregation use slightly different questions. But all have a fixed expected answer from a starting component name. E.g., who is the owner of Application <app-name>? |
Access | Ardoq Assistant, Ardoq MCP |
Retrieval toolsets | neighbours, reports, viewpoints |
Asset size | narrow / broad × sparse / dense |
Model | Claude Sonnet 4.6, Opus 4.6, Opus 5.5 |
Held fixed | Value |
Metamodel tools | available in every run; not under test |
Scoring rule | the same for Assistant and MCP |
Model configuration | the Assistant's defaults, in both Assistant and MCP (see Models) |
Assistant and MCP: two harnesses, run separately
The benchmark drives Ardoq AI through two harnesses that share the same goldens and the same scoring rule.
MCP harness: an agent that reaches Ardoq only through the Ardoq MCP server, with models served through Ardoq's LLM gateway. It represents an external AI agent using Ardoq through MCP. It is our own reference agent with its own system prompt, not any particular customer's MCP client. The answer comes back as a structured list of entities.
Assistant harness: the Ardoq Assistant, called through its API evaluation endpoint. The request pins which tools the Assistant may use and asks for a structured answer. It represents what an Ardoq user actually gets.
Why both are run, and why separately:
Tool differences: the two expose the same capabilities under slightly different tool names and schemas (for example
report_get_allover MCP,list_reportsin the Assistant).Different system prompts: the Assistant runs the product prompt; the MCP agent runs the benchmark's own.
Prompt caching: only the Assistant uses it, so token-based cost is not comparable between the two.
Agent loop and limits: retries, usage limits and error handling differ, which affects how a run ends when the agent cannot find an answer.
Model availability: each harness can only run the models deployed to it.
Answer quality is compared between the two on matched questions, models and toolsets (RQ4). Tokens and cost are never compared across them.
Models
Single-hop benchmarks use Claude Sonnet 4.6, Opus 4.6, Sonnet 5.5 and Opus 5.5 (or a subset) matching the models the product uses internally. Other models, including open-weight models, are evaluated as part of Ardoq's internal testing and model selection. Those results will be published for comparison purposes where relevant
The model configuration is the Assistant's default, and the MCP agent uses the same configuration:
Effort: High.
Thinking: the lowest level, which is off for the 4.6 models and the default adaptive level for the 5.5 models.
Keeping to the defaults keeps results simple to interpret and matches what users get. The settings may be adjusted if retrieval accuracy is not at an acceptable level.
Ardoq AI runs primarily on the Anthropic 5.5 family of models (at the time of this benchmark). But was primarily using the 4.6 family until recently. Models in between did not show a significant improvement in price/performance until 5.5.
Retrieval toolsets
Every run gets the same base tools: metamodel tools, component and reference search, field lookup and entity resolution. On top of those, exactly one retrieval toolset is under test:
Toolset | Tools (MCP / Assistant) | Retrieval strategy |
neighbours |
| Walk the graph outward from a named component |
reports |
| Find and read a tabular Ardoq report |
viewpoints |
| Follow a pre-defined traversal of the graph |
Write and population tools are excluded from every configuration: the benchmark is read-only.
Under evaluation: sparql. A fourth toolset in the Assistant answers questions by querying the graph with SPARQL rather than selecting Ardoq assets. It is Assistant-only, and will be evaluated when it becomes available for a production-quality test.
Asset size and curation
AI retrieval reliability is correlated with context size. As the context size increases, the quality of response decreases. So an asset, such as a report, that is extremely large (many rows and many columns) is likely to give a worse result than one which contains the same data in fewer rows and fewer columns. To test this, for each scenario, we create two versions of the reports and viewpoints. Both contain the information needed to answer the question but one has just enough information whilst the other has considerably more information than is needed.
The two viewpoint definitions below are used for the “Who is the owner of <name> application?”. The first is the ‘sparse’ viewpoint with just enough connections. The second is the ‘dense’ viewpoint with many superfluous connections. The terms ‘narrow’ and 'broad' represent the same distinction for reports.
The same question, with the same expected answer, is asked against reports and viewpoints of different sizes. Two independent dimensions describe an asset:
Dimension | Value | Meaning |
Breadth | narrow / broad | Few or many columns (reports), or traversal paths (viewpoints), beyond what the question needs |
Connectedness | sparse / dense | Few or many references and components connected to the entity being retrieved |
That gives four partitions: narrow-sparse, narrow-dense, broad-sparse and broad-dense. A curated asset is narrow-sparse (RQ3).
Asset size is controlled through permissions. Each partition has its own Ardoq evaluation user who can see only that partition's assets. The run's user token is the only thing that selects the partition, so the question, the toolset and the expected answer stay identical. Neighbours is not varied this way: its context depends on how connected a component is, not on asset permissions.
The reports used to create goldens are hidden from every evaluation user, so the AI cannot read the answer key.
Questions and ground truth
Goldens (question and expected answer pairs) are generated from the test data, not written by hand. For retrieval, what matters is the structure of the data, the metamodel shape, rather than particular names.
Query creation: the question is written as a Gremlin query and checked against a copy of the graph.
Golden query report: gremlin query generates a report with the complete set of question/answer pairs. Running the report deterministically produces the correct answers.
Generate goldens: the report's rows are turned into goldens in the production AI eval infrastructure.
Every question type has two parts on the same metamodel path:
Retrieval: a valid answer exists and must be found.
Negative rejection: no valid answer exists, and the AI must say so rather than invent one.
Question catalogue
# | Question type | Metamodel path |
1 | Single-hop |
|
2 | Two-hop (business capability) |
|
3 | Three-hop (application integration) | Who are the owners of the Applications integrated with this Application two paths via Interface (see below) |
1. Single-hop
The simplest possible retrieval, establishing a baseline before multi-hop queries. Both parts share the metamodel path {Person, Owns, Application}. This schema shape is a requirement of the golden data, not something being tested.
Retrieval. "Who is the owner of the {input} application?", where {input} is an Application that has an owner. The answer is scored against the owner fixed at golden-construction time.
Negative rejection. The test data includes Applications with no incoming Owns reference. A report listing them generates goldens with the same question template. The expected answer for every golden in this set is empty (no owner), and the AI is scored on reporting that no owner exists rather than inventing one.
2. Two-hop (business capability)
Chains two metamodel relationships: {Business Capability, Is Realized By, Application, Owns, Person}. Each Business Capability is realized by a set of Applications, some owned by a Person and some not.
Retrieval. A report lists, per Business Capability, the realizing Applications that have an owner, and those owners. The question is "Who are the owners of the applications that realize the '{capability}' business capability?". A capability can be realized by several owned applications, so this also tests whether the AI retrieves the full set of owners rather than stopping at the first.
Negative rejection (design TBD). The current design uses a second report listing, per Business Capability, the realizing Applications with no owner, with the same question template. The expected answer is the list of those Applications, so that the AI must surface that an application lack an owner rather than omit them or invent owners. This design is under review.
3. Three-hop (application integration)
“Who are the owners of the Applications integrated with {input} application”. This is harder than question type 2 not because of hop count, but because the answer requires reconciling two independent metamodel paths that lead to the same Application:
{Application, ardoq_parent, Interface, Connects To, Application, Owns, Person}{Application, Connects To, Interface, ardoq_parent, Application, Owns, Person}
Retrieval. "Who are the owners of the applications that are integrated with the {input} application?". Correct owners can be reachable through either path, so a toolset that follows only one direction of the Interface relationship will systematically miss results.
Negative rejection (design TBD). Constructed like question type 2's negative set: a report lists integrated Applications (via either path) with no owner. This design is under review.
Further question types (multi-hop, aggregation, and beyond) will be added here as they are designed.
Scoring and metrics
Accuracy
The answer is a structured list of entity names, compared with the expected names as sets.
Names are normalized (case and whitespace) and then matched exactly. A near-miss such as "J. Smith" vs "John Smith" counts as both missing and unexpected.
The score is set F1, so a partial answer earns partial credit (3 of 5 expected names, none spurious: precision 1.0, recall 0.6, F1 0.75).
A case passes only on an exact match (threshold 1.0).
Both are reported:
pass_rate(the strict gate) andmean_score(mean F1). Without the second, "missed one owner of five" is indistinguishable from "returned nothing".Pass rates carry 95% Wilson ranges. Overlapping ranges are read as "not distinguishable".
Negative rejection: on a question with no valid answer, returning nothing scores 1.0. Any name returned is wrong.
Reliability: consistency of the answer across repeated runs of the same question and configuration.
Aborted cases: a run that errors or hits a usage limit scores 0. It is never treated as a correct "no answer". Aborted cases are counted separately and left out of averages.
Context: three token figures are kept separate.
Answering context: the prompt size when the model gave its answer. This is the figure the context research question is about.
Cumulative tokens: the total across all model requests. Each turn of a tool loop re-sends the conversation so far, so this measures cost, not context.
Peak context: the largest single prompt. This checks whether history was dropped mid-run.
Cost and latency
Tool calls and duration per case.
Estimated cost in tokens only, because prompt caching makes price-based estimates misleading.





