In today’s world data is everywhere. We are surrounded by it in the form of spreadsheets, sales figures, energy consumption, personnel data, budgets, etc. Market data drives company strategy, production data informs about manufacturing processes and costs, and sales figures contribute to profitability. Data is essential and the lifeblood of business.

Understanding data, categorizing it, using it, analyzing it, tracking it, storing it, and deriving conclusions from it are therefore critical for the success of any organization.

This is where data lineage becomes important.

Data lineage is, essentially, a map of the journey data takes through a company. It starts with its origin, references each process and transformation along the way, including an explanation of how and why the data has moved over time, through to its delivery to an endpoint or end-user.

Typically Data Lineage can be split into two, distinct, types:

Solution data lineage is, therefore, more transactional/process related whereas Enterprise data lineage focuses on the high-level and bigger picture. This primer will focus on Enterprise data lineage.

An important part of data lineage is understanding how the data exists within your technical- and business landscape. There are three ways or “master data management patterns” in which data is used and moved around within an organization:

These are explained in more detail in our Data Lineage Metamodel document.

In some instances, data lineage can be documented visually to show the source of the data, each process or change it encounters on its journey within the organization, and its destination.

As the volume of data streams grows, especially via the cloud, it is becoming ever more critical for organizations to fully understand the data lifecycle: mapping how and where data is sourced, its transformation, and where it is stored. This knowledge is becoming increasingly important from a compliance and governance perspective.

Data governance requires that data be stored and processed in line with organizational policies and, importantly, regulatory standards.

Data lineage can be used to provide compliance auditing, improve risk management, and ensure data is stored and processed as required. An example of this is GDPR compliance, especially in the processing and storage of data.

Understanding everything about your data including its provenance, what information it actually provides, how your organization processes it, and where it finally resides is critical for a number of reasons.

Specifically, data lineage will help your organization with the following:

As the volume of data, together with societal attention on ethical data usage, increases it is essential that businesses know how and where their critical data comes from, is processed, and stored.

Data lineage provides this information through two streams: Enterprise data lineage, which provides a high-level view of the data, how it flows between applications (the patterns), and where it is stored, and Solution data lineage, which provides a lower-level, transactional, view of the data, where it comes from and how it is processed.

As compliance and governance directives become stricter having knowledge about data flows is essential.

For more detailed information about how Ardoq deals with data lineage please read our article on data lineage metamodel.