Why we benchmark
The quality and reliability of AI answers are becoming a key strategic differentiator for Ardoq, but "the AI is good" is a claim, not evidence. This initiative measures AI quality with consistent metrics that can be tracked across model versions, tool strategies and product changes.
The benchmarks serve three audiences:
Internal: evidence for product decisions, such as which models the Assistant runs, which tools it exposes, and whether a product change improved or degraded answer quality. Re-running a benchmark before and after a change works as a regression check.
Customers: quantified, reproducible guidance for both Ardoq Assistant users and customers who connect their own AI to Ardoq through MCP. Our Context Engineering article argues that tool choice, ontology design, query type and context construction all affect answer quality. Benchmarks turn those qualitative claims into measured ones.
Industry: position Ardoq against published AI benchmarks (for example MRCR v2 and GraphWalk) using comparable measurements so others can verify Ardoq’s AI quality. Our goal is for products in our space to collaborate on an industry benchmark that simplifies comparing AI quality and reliability in our domain.
How Ardoq benchmarks are built
These principles apply to every benchmark below. Each benchmark's own page describes how it applies them.
Ground truth comes from Ardoq data. Expected answers are produced by deterministic queries against the test data, not written by hand. "Correct" means "what the data actually says".
Scoring is deterministic wherever possible. Scoring is done in our production AI eval infrastructure, which uses code-based comparisons where possible. An LLM judge is used only where no exact answer exists.
Models match the product. Benchmarks use the models that Ardoq AI uses or plans to use, at their default settings. Other models, including open-weight models, are evaluated as part of Ardoq's internal testing and model selection. Those results may not always be published.
Common metrics: accuracy, reliability across repeated runs, negative rejection (correctly saying "there is no answer"), plus cost, latency and tool use.
Stated limits. Every result page states what its data can and cannot support: which question types, models and configurations it covers, how many repeats, and where a result is a correlation rather than a cause.
Benchmarks
Below are the benchmarks that are completed, in-progress, or planned. If you’d like to see other benchmarks, leave your suggestion on the Product Portal card.
Benchmark | Question it answers | Status |
Can Ardoq AI reliably find entity data in Ardoq, and what affects that? | Running | |
Algorithmic / quantitative analysis | Can Ardoq AI compute rankings, aggregations and statistics correctly? | Planning |
Time-based queries | Can Ardoq AI reason about history and change? | Proposed |
Inference | How much do metamodel and ontology descriptions and constraints improve answers? | Proposed |
Technique execution with and without rubrics | Does a rubric make EA techniques more consistent? | Proposed |
AI productivity | Is there a measurable productivity gain for architects? | Proposed |
Professional reasoning, adapted to EA | Can existing professional-reasoning benchmarks be repurposed for EA? | Proposed |
Whether Ardoq AI reliably retrieves entity data from Ardoq, and how that depends on the retrieval toolset, the size of the assets it reads, the model, and whether it is reached through the Assistant or MCP. See Retrieval Benchmarks.
Algorithmic / quantitative analysis
LLMs are notoriously unreliable at arithmetic and algorithmic expressions. This benchmark tests our techniques for performing basic quantitative analysis over Ardoq data, for example "top 10 most expensive applications" or "the median risk value across all applications in this workspace". It measures how accurately and reliably the AI computes correct numeric answers (rankings, aggregations, statistics) rather than approximating or hallucinating them.
Time-based queries
How well an agent handles questions with a temporal dimension, such as "what changed in this workspace since last quarter" or "which components were added after the migration". This depends on Ardoq's history and versioning data being surfaced correctly, and on the agent reasoning about date ranges and change events rather than only the current state.
Inference
How much the answer quality depends on the semantic scaffolding around the raw graph data. The same query set is run with increasing levels of context:
without metamodel descriptions
with metamodel descriptions
with ontology descriptions
with ontology rules & constraints
The goal is to isolate how much accuracy comes from giving the AI model an explicit model of what component and reference types mean and how they may legally relate, versus leaving it to infer that from names and structure alone.
Technique execution with and without rubrics
Whether providing a scoring rubric or worked methodology for an EA technique (for example application rationalization or risk assessment) improves the consistency and quality of an agent's execution of that technique, compared with reasoning unaided.
Begin with well-defined techniques, such as disaster recovery analysis.
Then extend to other techniques and agent frameworks.
Use techniques shown to improve external professional reasoning benchmarks
AI productivity
Can we show quantitatively that AI assistance makes architects more productive, not just more able to get an answer?
Follow approaches such as METR's Expenditure Horizon, which measures the length and complexity of task an AI system can complete reliably as a proxy for capability growth over time.
Adapt it to EA tasks to show a measurable productivity curve for Ardoq AI, rather than relying on anecdotal customer feedback.
Professional reasoning, adapted to EA
Review existing benchmarks designed to test professional or domain reasoning (as opposed to general knowledge retrieval or coding), and assess whether their structure and scoring can be repurposed for enterprise architecture reasoning tasks.
