Skip to main content

Ardoq AI Benchmarks

Introduction to Ardoq's AI benchmarks and links to specific results

Written by Jason Baragry

Why we benchmark

The quality and reliability of AI answers are becoming a key strategic differentiator for Ardoq, but "the AI is good" is a claim, not evidence. This initiative measures AI quality with consistent metrics that can be tracked across model versions, tool strategies and product changes.

The benchmarks serve three audiences:

  • Internal: evidence for product decisions, such as which models the Assistant runs, which tools it exposes, and whether a product change improved or degraded answer quality. Re-running a benchmark before and after a change works as a regression check.

  • Customers: quantified, reproducible guidance for both Ardoq Assistant users and customers who connect their own AI to Ardoq through MCP. Our Context Engineering article argues that tool choice, ontology design, query type and context construction all affect answer quality. Benchmarks turn those qualitative claims into measured ones.

  • Industry: position Ardoq against published AI benchmarks (for example MRCR v2 and GraphWalk) using comparable measurements so others can verify Ardoq’s AI quality. Our goal is for products in our space to collaborate on an industry benchmark that simplifies comparing AI quality and reliability in our domain.

How Ardoq benchmarks are built

These principles apply to every benchmark below. Each benchmark's own page describes how it applies them.

  • Ground truth comes from Ardoq data. Expected answers are produced by deterministic queries against the test data, not written by hand. "Correct" means "what the data actually says".

  • Scoring is deterministic wherever possible. Scoring is done in our production AI eval infrastructure, which uses code-based comparisons where possible. An LLM judge is used only where no exact answer exists.

  • Models match the product. Benchmarks use the models that Ardoq AI uses or plans to use, at their default settings. Other models, including open-weight models, are evaluated as part of Ardoq's internal testing and model selection. Those results may not always be published.

  • Common metrics: accuracy, reliability across repeated runs, negative rejection (correctly saying "there is no answer"), plus cost, latency and tool use.

  • Stated limits. Every result page states what its data can and cannot support: which question types, models and configurations it covers, how many repeats, and where a result is a correlation rather than a cause.

Benchmarks

Below are the benchmarks that are completed, in-progress, or planned. If you’d like to see other benchmarks, leave your suggestion on the Product Portal card.

Benchmark

Question it answers

Status

Can Ardoq AI reliably find entity data in Ardoq, and what affects that?

Running

Algorithmic / quantitative analysis

Can Ardoq AI compute rankings, aggregations and statistics correctly?

Planning

Time-based queries

Can Ardoq AI reason about history and change?

Proposed

Inference

How much do metamodel and ontology descriptions and constraints improve answers?

Proposed

Technique execution with and without rubrics

Does a rubric make EA techniques more consistent?

Proposed

AI productivity

Is there a measurable productivity gain for architects?

Proposed

Professional reasoning, adapted to EA

Can existing professional-reasoning benchmarks be repurposed for EA?

Proposed

Whether Ardoq AI reliably retrieves entity data from Ardoq, and how that depends on the retrieval toolset, the size of the assets it reads, the model, and whether it is reached through the Assistant or MCP. See Retrieval Benchmarks.

Algorithmic / quantitative analysis

LLMs are notoriously unreliable at arithmetic and algorithmic expressions. This benchmark tests our techniques for performing basic quantitative analysis over Ardoq data, for example "top 10 most expensive applications" or "the median risk value across all applications in this workspace". It measures how accurately and reliably the AI computes correct numeric answers (rankings, aggregations, statistics) rather than approximating or hallucinating them.

Time-based queries

How well an agent handles questions with a temporal dimension, such as "what changed in this workspace since last quarter" or "which components were added after the migration". This depends on Ardoq's history and versioning data being surfaced correctly, and on the agent reasoning about date ranges and change events rather than only the current state.

Inference

How much the answer quality depends on the semantic scaffolding around the raw graph data. The same query set is run with increasing levels of context:

  • without metamodel descriptions

  • with metamodel descriptions

  • with ontology descriptions

  • with ontology rules & constraints

The goal is to isolate how much accuracy comes from giving the AI model an explicit model of what component and reference types mean and how they may legally relate, versus leaving it to infer that from names and structure alone.

Technique execution with and without rubrics

Whether providing a scoring rubric or worked methodology for an EA technique (for example application rationalization or risk assessment) improves the consistency and quality of an agent's execution of that technique, compared with reasoning unaided.

  • Begin with well-defined techniques, such as disaster recovery analysis.

  • Then extend to other techniques and agent frameworks.

  • Use techniques shown to improve external professional reasoning benchmarks

AI productivity

Can we show quantitatively that AI assistance makes architects more productive, not just more able to get an answer?

  • Follow approaches such as METR's Expenditure Horizon, which measures the length and complexity of task an AI system can complete reliably as a proxy for capability growth over time.

  • Adapt it to EA tasks to show a measurable productivity curve for Ardoq AI, rather than relying on anecdotal customer feedback.

Professional reasoning, adapted to EA

Review existing benchmarks designed to test professional or domain reasoning (as opposed to general knowledge retrieval or coding), and assess whether their structure and scoring can be repurposed for enterprise architecture reasoning tasks.

Did this answer your question?