Introduction
This article shows how to use Ardoq in combination with a Large Language Model (LLM) for Retrieval Augmented Generation (RAG). Specifically, it will use the LLM to automatically suggest relationships between an Application and the Business Capabilities it realizes and then add those to Ardoq through the API. The suggestions will be based on Business Capabilities that are already defined in Ardoq rather than newly created by the LLM.
The average organization has over 500 applications. Many of our customers are in the 1000s. This solution can minimize the amount of effort needed to add connections to that ever changing list. Ideally, it would be a solution that whenever a new Application is detected, for instance when imported from a CMDB, then a suggested set of relationships would automatically be created for the Business or Technical Capabilities it realizes. The automation would be built-in to the application governance playbook.
This article explores the underlying mechanism for that automated solution so that you can explore similar solutions yourself. An extended approach would make the functionality interactively available through the UI. The purpose of this article is to illustrate the potential of LLM RAG pipelines using Ardoq rather than a full, production-ready solution.
This article describes the RAG approach and the details of including Ardoq in a RAG pipeline. If you are only interested in the result then jump to the section on Testing the Solution.
Viability of the solution
The goal of this article is to show the basic mechanics of including Ardoq in a RAG pipeline. Nevertheless it's interesting to explore the viability of the example problem - automatic inference of relationships between Applications and Business Capabilities. Especially the potential benefit, the technical feasibility, and the confidentiality issues.
As the industry gets over the initial hype of GenAI it is focusing more on return on investment from these solutions. There is a need to reduce the effort and improve the quality of enterprise architecture model creation and maintenance.
Business Capabilities are a kind of general knowledge that LLMs should have as part of their training. A capability map will also include description information from the organization that uses or provides the capability map.
Capability maps are small enough that they will fit within the context window of an LLM with minimum effort. See the section on Context Window below for more detail.
Descriptions of commercial packaged applications (COTS and SaaS) are also freely available information on the internet that we can assume LLMs have as part of their training. Home-developed systems can include enough description to allow the inferencing needed.
One area of concern is confidentiality. The link between applications and capabilities isn’t (usually) business specific IP. The application list of an organization however is usually considered confidential because it can identify known vulnerabilities and provide an attack surface for bad actors.
Confidentiality is a key criteria in how Ardoq is designing its solutions in this area. We recommend consulting the privacy notice of your LLM provider before passing this kind of information yourself. The approach in this document can be used with public SaaS LLMs, enterprise-licensed SaaS LLMs, and open source LLMs that you host yourself.
What is Retrieval Augmented Generation (RAG)?
RAG is a technique that combines the LLM with an external knowledge source to overcome some of the known shortcomings of LLMs for generative AI. These LLM shortcomings include:
The gap between the most up-to-date information and the information available at the time the model was trained
Hallucinations. The generation of fictitious information.
The inability to leverage your own content in GenAI
The inability to perform data lineage on the generated results
RAG allows the model to incorporate specific data that was not included in the original training to overcome these limitations and ensure lineage of the information generated.
For our objective we want to utilize the categorization capabilities of LLMs to suggest relationships between an Application and the Business Capabilities it may realize. However, we want it to only use the Business Capability Model of the organization that is captured in Ardoq rather than suggest its own.
The typical RAG application combines the user’s query with information from an external source to form the context for the request to the LLM. The result from the LLM is then more precise and pertinent for the user.
Using a knowledge-graph like Ardoq can significantly simplify the Indexing and Retrieval stages of RAG. The Indexing and Retrieval steps shown are needed to deal with the context window limitations of LLMs. That is, the limited amount of information that can be provided in the query request. To meet the constrained context window, traditional RAG applications must chunk external knowledge sources into small pieces and then identify the most relevant pieces to include with the user’s query in the LLM request. This is often done with embeddings and vector databases:
Indexing: Splitting the knowledge source into shorter chunks (embeddings) and building a vector database..
Retrieval: Retrieving relevant information fragments based on the similarity between the user query and the chunks.
Knowledge-graphs are already highly structured and allow retrieval of information based on defined relationships. Additionally, they allow us to make associations in the data that might not otherwise be derivable from embedding based retrieval. Therefore, in this scenario, we can forego the usual steps of chunking information and identifying relevant pieces using embeddings.
Alternatives for building a RAG pipeline with Ardoq
There are multiple ways a RAG pipeline can be configured to solve our problem. This list is not exhaustive.
Send the entire capability map from Ardoq in the request context.
Augment the retrieval step by selecting a smaller subset of the capability model to include in the request.
Configure the LLM to use the entire capability map as an external knowledge source before sending requests. This approach uses the capabilities of LLM providers to automatically apply indexing and retrieval techniques in the background
Build a traditional RAG pipeline that combines the knowledge-graph with embeddings and vector-based retrieval.
This article will take the simplest approach and include the capability map in the request context to OpenAIs gpt-4 model. The benefit of this approach is that it can be applied to multiple commercial and open-source LLMs. A subsequent article may explore ways to augment retrieval by making a smaller selection from Ardoq to include.
If you are interested in the alternatives, OpenAI Assistants is one tool that allows you to preconfigure the external knowledge source for an LLM to implement RAG. Indeed, this example problem of capability realization has also been tested with OpenAI Assistants.
There are numerous examples available for building a traditional RAG pipeline using embeddings and vector-based retrieval. This cookbook combines OpenAI with a graph database (Neo4j) and is orchestrated using Langchain. It could easily be adapted to use Ardoq.
Ensuring our Capability Map fits the Context Window
A prerequisite for sending the entire capability map is ensuring it will fit within the context window of the LLM. This article uses the OpenAI GPT-4 Turbo model which supports a context window of 128,000 tokens. Let's see how many tokens are in the capability map.
The capability map is the sample business capability map provided in the Business Capability Modelling Use Case Solution. It consists of 290 capability components with names and descriptions.
The code samples in this document use the latest v2 Ardoq API. The sample v2 api python client is included in the ardoqpy python package.
from ardoqpy.ardoqpy_v2 import API
import tiktoken # openai utility to count tokens
...
encoding = tiktoken.encoding_for_model('gpt-4-1106-preview')
tokens = encoding.encode(bizcaps_s) # string for the capability map
len(tokens)
(a subsequent code block will show how the bizcaps_s capability map was extracted)
This gives 24718 tokens, which is a significant number of tokens, but still within the 128k limit. Other LLMs have a smaller window to work with so check before recreating this example.
It's necessary to note that research shows LLMs show less performance when dealing with large context windows. They place more emphasis on information at the beginning and end of the prompt and less on the information in the middle. Nevertheless, the performance is ok for this RAG example.
Retrieving the Capability Map and relevant Applications from Ardoq
Two pieces of information need to be extracted from Ardoq to be included in the query:
The Business Capability Map
The list of Applications that do not have an is Realized By relationship from a Business Capability
Retrieving the capability map requires extracting all capabilities, filtering out unnecessary information to minimize the context window, and maintaining the capability hierarchy so references can be made to the lowest level capability that is applicable.
The capabilities in our map contain numerous bits of information to track attributes such as Market Differentiation, Complexity, Current Maturity Level, and Attributed Cost. These will be filtered out during retrieval to only leave:
ID
Name
Description
Parent ID
The Application should be related to the lowest level Business Capability in the hierarchy that is applicable. Therefore the LLM must understand the hierarchy of the capability map. Testing shows that LLMs can infer the hierarchy of the capability map if the Parent ID field is supplied and the information is described as hierarchic. To check this you can pass a set of capabilities to an LLM and ask it to print the result in a hierarchy.
The latest version of OpenAI gpt (at the time of writing) is able to reason in terms of JSON structures. This feature is used to make it easier to parse information between the Ardoq API and OpenAI.
Retrieving and filtering the capability map
Retrieving the capabilities is straightforward with the list components endpoint in the API.
First you need to find the workspaceID for your capability map. This is available in the URL field if you open the workspace.
Then you call the workspace context endpoint to retrieve information about componentTypes and their IDs. Also take note of the referenceTypes because we’ll need that to create the correct reference type in Ardoq when adding the generated relationships.
import json
from ardoqpy.ardoqpy_v2 import API
import openai
... # token and ID strings ommitted
ardoq = API(
ardoq_api_host=ardoq_host,
ardoq_api_token=ardoq_token
)
bizcaps_query = {"rootWorkspace": bizcaps_ws, "typeId":
capability_compID}
components = ardoq.list_components(query_params=bizcaps_query)
bizcaps = []
for c in components['values']:
bizcaps.append({key: c[key] for key in ['_id', 'name',
'description', 'parent']})
bizcaps_s = json.dumps(bizcaps, indent=2)
Retrieving the list of applications that need connections to business capabilities
We only want to retrieve the Applications which do not have an existing reference from a Business Capability. This is possible to do by creating a report in Ardoq which lists only those Applications and then extract the details with the run report API endpoint.
The report will need to include:
ID (so we can use it to create the reference to this component later)
Application Name
Description
For known packaged applications, such as Salesforce, we can assume the descriptions have been included in the public internet training materials for the LLM. For internally developed applications the description field will need to contain enough information for the LLM to match it to relevant capability descriptions.
The Business Capability Realization Use Case Solution already includes a report that shows Applications not connected to a business capability:
BCR DQ - Application not Realizing any Business Capabilities
Simply extend the definition of this report so that it also includes ID and description fields
This report uses gremlin - find all nodes of type ‘Application’ which do not have an incoming ‘is Realized By’ reference from a node of type ‘Business Capability’.
Recent changes have made it possible to create the same report without gremlin.
This report definition lists all Applications that do not have a reference to either a Business Capability or Technical Capability. You may want to limit this to simply Business Capability component types depending on how you use Business and Technical Capabilities.
View the report you have just created to find the ID for the report.
The code to extract the report to a JSON object:
apps_without_caps = ardoq.report_run(id=report_id, view="objects")
We can now iterate through the apps_without_caps[‘values’] to send the request to the LLM. To create a string that can be passed to the LLM based on the JSON object from Ardoq you could use:
app_s = json.dumps(apps_without_caps[‘values’][0], indent=2)
Prompt Design and Generation
Prompt design is critical to getting accurate and consistent results from an LLM. Start with a system message that gets gpt-4 to output JSON data.
system_msg = "You are a classification assistant designed to output JSON"
The user prompt starts with instructions for the query. It specifies:
How the capability map information is structured
How each capability is described
How the application is described
That the LLM should only use business capabilities that are provided. It should not generate its own.
assistant_prompt=(\
"I will provide a JSON list of business capabilities." +
"Capabilities are organized in a hierarchy." +
"Each capability has a name, ID, description, and the ID of its parent capability if it has a parent." +
"I will provide you with an application. The application is described in the JSON format 'name : description'" +
"If you know of the Application then you can also use your own description" +
"You will classify the application according to the business capabilities that it supports." +
"You will choose the capability at the lowest level in the hierarchy that matches." +
"An application can support multiple capabilities" +
"If you do not know the answer then say Not Known. Do not guess." +
"Only use capabilities from the information provided" +
"Present the results as a list of capabilities in JSON format"
)
Finally the request is constructed and passed to the LLM and the result is converted to JSON to add back to Ardoq.
user_msg = assistant_prompt
user_msg += "here is the json describing business capabilities. " + bizcaps_s
user_msg += "here is the application to classify. " + app_s
response = gpt_client.chat.completions.create(
model=gpt_model,
response_format={"type": "json_object"},
temperature=0,
seed=1,
messages=[
{"role": "system", "content": system_msg},
{"role": "user", "content": user_msg}
]
)
mapped_caps = json.loads(response.choices[0].message.content)
The LLM then returns a list of JSON dictionary items similar to this:
{
"capabilities": [
{
"_id": "edb3faa34652f7a5c06283f2",
"name": "Risk Policies and Framework"
}
]
}
Adding generated information back into Ardoq
The public API for adding this information back to Ardoq is the create reference endpoint.
Simply iterate through this suggested capabilities list and adding the references from the capability to the application using the IDs:
for cap in mapped_caps['capabilities']:
ref = {'type': isRealizedBy_refTypeID, 'source': cap['_id'],
'target': application_test['_id']}
ref_res = ardoq.create_reference(data=ref)
Testing the Solution
Here is the capability map which shows Capabilities and the Applications that realize them.
In this sample we have many applications supporting the Marketing Capabilities but nothing for the Contact Center capabilities.
We can also flip the capability map to show the Applications and the Capabilities that they realize.
The organization does have an application used for Call Center operations - Genesys Pure Connect. But it has not been connected to any Business Capabilities:
It has a simple description:
“Genesys PureConnect is a customer interaction center (CIC) for both on-premise & cloud (CaaS)”
Let's see if the LLM can suggest useful capability references. Here is the result of the call to the LLM:
2024-01-28 20:01:14,427 HTTP Request: POST https://api.openai.com/v1/chat/completions "200 OK"
{
"capabilities": [
{
"_id": "5ead817f5b95589aa3bea8f5",
"name": "Contact Center Operations",
"description": "The ability to operate and manage contact centers."
},
{
"_id": "ea4bf449accb0681932d9502",
"name": "Contact Center",
"description": "The setting up and management of a contact center in support of sales, service delivery, outbound marketing and general enquiries."
}
]
}
The capability map is automatically updated with the relevant references.
Note: The LLM suggested two capabilities - one of which is the parent of the other. This contradicts the prompt instruction. But this can be fixed with a little prompt manipulation.
Conclusion
We are exploring many ways that Ardoq can be improved using AI techniques and also how Ardoq can be used to help manage AI innovation and risk management within your organization.
This example is provided to show the basics of setting up a RAG pipeline with Ardoq. There are many areas to explore in terms of retrieval optimization, prompt engineering, and how to add information back to Ardoq that allows human approvals in the most effective way.
There is more work to do before this type of solution can be productized and there are many articles explaining the limitations that need to be overcome.
One issue is that we can’t always trust the results of LLM generated text. With an interactive solution - i.e., built into the UI of the tool - it is easy to validate the results. For an automated solution there needs to be validation mechanisms enabled. This could be implemented today using Broadcast notifications to application owners. Another approach is to provide approval flows or even create entire Scenarios for approval
Enterprise Architecture model design and population requires a considerable investment to ensure the dataset is large enough to allow business outcomes to be achieved and that the quality of data is maintained at a high enough level to trust in those results. GenAI and LLMs show promising results for helping with these steps.