Alation Integration

Overview

Alation shows people what data exists, who owns it, and where it flows. Validatar tests whether that data can be trusted. The Alation integration connects the two so that each platform shows what the other knows.

• Validatar to Alation. Trust scores, quality scores by dimension, and the test results you choose appear in Alation, on the tables and columns they describe.
• Alation to Validatar. Alation tags, domains, stewards, trust flags and lineage appear in the Validatar catalog, next to the tests that cover the same data.

The integration runs inside Validatar. It uses Alation's public APIs and Validatar's own scheduling, so there is no separate server, connector or scheduler to maintain.

Get the integration

Five items on the Validatar marketplace make up the integration:

What appears in Alation

Validatar publishes to Alation's Data Health feature, which Alation provides for third-party data quality tools. Each published item becomes a data health rule on the table or column it applies to, with a status of Good, Warning or Alert and a link back to the result in Validatar.

Once rules are published, Alation does the rest:

• The Health tab on a table shows the most severe status across its rules.
• Search results mark tables that have an Alert or Warning.
• Tables, BI data sources and reports downstream of a problem show it as an upstream issue, through Alation's own lineage.

Scores are published after each scheduled test run. Results for tests you mark as critical can be published the moment the test runs, using a test action.

Note: Alation keeps the current value of each rule. Score history and trends remain in Validatar, one click away through the link on each rule.

How statuses are set

• Scores are Good above a threshold you choose, Warning in a middle band, and Alert below it. When a table has an impact rating with a target score, that target can serve as the Good threshold for the table. Trust scores use the same 0 to 100 scale as quality and coverage scores: Validatar's 0 to 5 star trust score times 20, so 4.5 stars is 90. A table that Validatar has not scored yet gets no score rules until it has a score.
• Test results are Good when the test passes. A failed test shows as Alert or Warning, chosen per test. A test that errors (for example, a connection timeout) shows as Warning, because it says nothing about the data itself.
• Dimension scores are the share of a table's tests in that dimension that passed their latest run. Dimensions with no tests on a table are not published, so an untested dimension never looks like a failing one.

Choosing what to publish

Publishing is controlled in three places, from broad to narrow.

1. Publish scope. A single setting on the Alation data source chooses how much Validatar publishes:

2. Test-level fields. When tests are in scope, test owners decide test by test using the fields in an Alation Publishing section on the test, which the Alation Project Custom Fields item adds:

• whether the test is published to Alation
• the name Alation users see for it, so a test can appear as "Order totals reconcile to the general ledger" rather than its internal name
• whether a failure shows as Alert or Warning
• which Alation object the result belongs on, so a column-level test can be shown on its column

By default only tests that are explicitly opted in are published. This keeps the Alation Health tab readable on tables with many tests.

3. Table-level opt-out. A table can be excluded from publishing entirely. This is useful for staging tables that are tested but are not meant for catalog users.

What appears in Validatar

From Alation, the integration brings in:

• tags
• domain membership
• stewards
• trust flags (endorsed, warning, deprecated) and their reasons
• the Alation title, description and a link to the object in Alation
• any additional Alation custom fields you name, such as a data owner or criticality rating
• table-level lineage, and optionally column-level lineage

These land in an Alation section of the Validatar catalog custom fields on the same data sources, schemas, tables and columns, and as Validatar lineage. They sit in their own fields, so Alation values never overwrite descriptions or tags maintained in Validatar. You choose which fields come across, and for which schemas, at the top of the metadata script.

With this context in Validatar, teams can prioritise testing by domain or criticality, route failures to the right steward, and see which downstream objects a failed test affects.

How it works

The integration is packaged as an Alation data source template. It holds the credentials for both the Alation and Validatar APIs and the code every job shares. Each job runs through a standard Validatar feature, on a connection built from the template with the same Alation settings:

Because the publish step is itself a Validatar test, every publish has a pass or fail history and can alert like any other test.

Checking the health of the Alation instance

The Alation data source's own catalog is the Alation instance: each Alation object type (data sources, domains, glossary terms, policies, tags and more) is a schema, and each object is a table whose columns hold its properties. The Alation Instance Health Checks pack tests that catalog and the Alation API behind it. It covers credentials about to expire, failed or stale metadata extraction per data source, empty domains, duplicate glossary terms and tags, suspended stewards, deprecated tables that dashboards still read, and whether Validatar's own Data Health feed into Alation is current.

Trying it without Alation

Turn on Demo Mode on the Alation data source and every Alation call is answered by a built-in sample instance, a fictional company with a few deliberate problems. The object catalog, the health checks and the scripts all run against it, and nothing is sent to Alation.

Matching the two catalogs

Validatar and Alation identify tables differently, so each job pairs a Validatar data source with an Alation data source and matches schemas, tables and columns by name within the pair. The metadata and lineage scripts name their Alation data source at the top of the script; publishing uses a connection map on the Alation data source. The Alation data source's own catalog lists every Alation data source it can see with a suggested Validatar match, so the pairs are built from evidence. Once the metadata script has run, each table's Alation key is stored in Validatar and publishing uses it directly. Objects found on one side and not the other are counted and reported, never silently dropped.

Requirements

Alation

• Alation Cloud Service, or a customer-managed Alation release that includes the Data Health API
• The Enable Health Data setting turned on by an Alation Server Admin
• An integration identity: an OAuth client application (Alation Cloud Service) or a service user with a refresh token
• An Alation role that can publish data health rules, read domains and read lineage. In most organisations this is a Server Admin role assigned to a dedicated system user

Validatar

• A Validatar API token for the integration
• Govern Edition, to define the integration's custom fields
• The Data Lineage feature, to bring in Alation lineage
• The Standard Test Actions feature, to publish critical test results the moment they run

Both platforms communicate over HTTPS. No Data Agent is needed.

Rollout

Most teams adopt the integration in stages. Each stage is useful by itself.

1. Prepare. Agree the decisions below, create credentials on both sides, and map data sources.
2. Publish scores. Add the publish step to the daily test job with Trust Score scope. Tested tables in Alation now show Validatar scores.
3. Publish dimensions and tests. Import the project custom fields, widen the scope, opt in the tests that matter to catalog users, and add test actions to critical tests.
4. Bring in Alation metadata. Import the Alation custom fields, then add the metadata script to each warehouse data source.
5. Bring in lineage. Start with table-level lineage, then add columns where needed.

Decisions to make before setup

Related

• Alation Data Source Template
• Alation Instance Health Checks
• Alation Metadata and Lineage Scripts
• Alation Catalog Custom Fields
• Alation Project Custom Fields
• What Is a Data Source Template?
• What are Test Actions
• How Trust Scores Are Calculated
• Quality Dimensions
• Impact Ratings
• Configuring Custom Fields

Note: This article describes the design of the Validatar and Alation integration. It has been built from Alation's published API specifications and tested in Validatar against a built-in sample Alation instance, including both ways of publishing: the publish step in a test job and the test action on a single test. It has not yet been validated against a live Alation environment. Setup details may change after that validation. To discuss using it, contact your Validatar account team.