Atlan Data Source Template

Overview

The Atlan data source template connects Validatar to Atlan in both directions from a single data source. That data source holds the credentials for both the Atlan and the Validatar APIs, and everything in the Atlan integration runs on its connection:

• Publishing to Atlan. Two macros publish Validatar trust scores, a score per quality dimension and the test results you choose to Atlan. By default they write the Validatar custom metadata set, which Atlan shows on each table with green, yellow and red badges. A preview option also writes Atlan's external data quality structure.
• Cataloging the Atlan tenant. Ingesting the data source's metadata turns Atlan's own objects (connections, glossaries and terms, tags, custom metadata, data domains and products, personas, purposes, groups and recent workflow runs) into Validatar tables, which the Atlan Instance Health Checks pack tests.
• Pulling from Atlan. The Atlan Metadata and Lineage Scripts run as Custom Metadata and Lineage Ingestion sources on connections built from this template, with the same Atlan settings. What is specific to each warehouse sits at the top of each script. The metadata script fills the Atlan Catalog Custom Fields.

Marketplace: Atlan - Data Source Template

How It Works

• One template, several hosts. The template's pre-execution script holds the Atlan client, the Validatar client and the code the metadata and lineage scripts share. It runs before every script on a connection built from the template: the data source's own metadata ingestion, the two macros, tests on the data source, and the Custom Metadata and Lineage Ingestion sources that use the template.
• Built on Atlan's Python SDK. Every Atlan call goes through pyatlan, Atlan's official Python SDK, pinned to version 11.4. pyatlan is loaded only when Demo Mode is off, so the template imports and runs in demo mode on any Validatar instance without it. To connect to a real tenant, install pyatlan==11.4.* once into the Python that runs Validatar scripts: under Settings > Configuration (the scripting Python modules) for the Validatar server or cloud runtime, or through the Data Agent's Python module install when the data source runs on a Data Agent. Keep the pin, because pyatlan renames attributes between minor versions. On a Data Agent, restart the agent service after the install: a running agent keeps the Python packages it has already loaded, and pyatlan does not import until it restarts. On a Data Agent, the first script after the agent starts takes about seven seconds longer while pyatlan loads; later scripts reuse the loaded SDK.
• Atlan objects become tables. Metadata ingestion catalogs the Atlan tenant itself. Each object type is a schema, each object a table named <name> [<id>], and each of the object's properties a column whose comment holds the current value. Only the Standard Fields every Validatar instance ships are written (friendly name, description, tags), so the object catalog needs no custom field setup. Validatar compares table names without regard to case, so when two objects would share a name, such as the glossary terms "Order" and "order", the second one gets its full id.
• Nothing is written to Atlan unless you ask. Ingestion and the health checks only read. Publishing writes three things, and nothing else: the values of the Validatar custom metadata set on matched tables, the Validatar entry of the external data quality structure when that target is chosen, and announcements whose title starts with Validatar:. Dry Run is on by default.
• Demo mode. With Demo Mode on, every Atlan call is answered from a built-in sample tenant: a fictional company with four connections (Snowflake PROD, Snowflake SAMPLE_DW, PostgreSQL CRM and Tableau Cloud) whose metadata is healthy in most places and has a few deliberate problems. Use it to try the template, the health checks and the scripts before Atlan credentials exist. Nothing is sent to Atlan while it is on, and pyatlan is not needed.

Authentication

Atlan supports two sign-in methods, chosen with Atlan Auth Method:

What the Atlan identity needs. Personas decide which connections the integration can read and update. A token or client without personas sees only the connections it created. Within those connections the integration needs:

• read access to assets and lineage, for ingestion, the health checks and the metadata and lineage scripts
• permission to update custom metadata values (the Validatar set), for publishing
• permission to update assets, for the external data quality target and for announcements
• an admin role to list API tokens and users, which two of the health checks do

An object type or check the identity may not read is reported as not permitted rather than passing. Permission errors fail at once: the template retries throttling (HTTP 429) and server errors up to four times with back-off, but never a 401 or 403, so a missing persona shows on the first run.

Request limit. Atlan allows 400 API requests a minute for the whole tenant and blocks for a minute when that is exceeded. Atlan Requests Per Minute (300 by default) caps what one script sends, leaving room for other tools on the tenant.

Validatar. An API token sent as the x-val-api-token header, with the Catalog, Authoring and DataSteward scopes. Publishing and the lineage script need it. The object catalog, the health checks and the metadata script do not, although with a token the Connection schema of the object catalog also suggests which Validatar data source matches each Atlan connection.

Connection Parameters

Pairing data sources with Atlan connections

Atlan names each connection default/<connector>/<number>, for example default/snowflake/1700000001. To publish a Validatar data source, the template needs the connection that holds its tables in Atlan. It reads that pairing from two places:

• The data source's Atlan Connection Qualified Name field. The Atlan metadata script writes the connection's qualified name to this field on the warehouse data source every time it runs, so a data source that runs the script is already paired. To pair a data source by hand, set the field on its catalog page. The field is part of the Atlan Catalog Custom Fields.
• The Atlan Connection Map parameter on the Atlan data source.

The map is read leniently. Commas and new lines also separate entries, : works in place of =, backslashes count as slashes (12=default\snowflake\1700000001), and the qualified name of a schema or table is cut back to its connection. An entry that still cannot be read is ignored, and the publish test's output names it. A field value that is not a connection qualified name is skipped and named the same way.

After the first metadata ingestion, the Connection schema of the Atlan data source lists every connection the integration can see, with its qualified name, its host, and the Validatar data source it is paired with, from either place. An unpaired connection shows a suggested Validatar data source when one has the same host. Copy qualified names from those rows.

Capabilities

Metadata Ingestion: the Atlan Object Catalog

Workflow runs are read from Atlan's app runs. A tenant that still runs legacy workflows may list none; this has not yet been checked on a live tenant.

Links to Atlan objects are built as <Atlan Base URL>/assets/<guid>/overview. Atlan does not document this path, so open one link after the first real ingestion to confirm it on your tenant.

Macros

What the publish test returns. One row per table and target, with these columns:

The test's script output starts with one summary line: tables considered, published, unmatched, skipped, without scores and left out by the scope parameters. Below it come the scope in force, each data source's Atlan connection and where that pairing came from, and anything that was ignored, such as an unreadable map entry or a scope entry that matched nothing.

Because the publish step is a Validatar test, every publish has a pass or fail history and can alert like any other test. Leave Dry Run on until the rows show what you expect.

How the test action works. A test action receives the test's URL and result, not its table. The action reads the test's Atlan Object Qualified Name field, which the publish test fills the first time it publishes with Dry Run off (or which you set by hand), and republishes that whole table so its scores, counts and failing-test list stay consistent. The table's trust score may lag by one scoring cycle; the test results and dimension scores are current. Until the field is filled the action reports that it skipped the test. It acts on tests that are opted in, or on any test when Post Announcements is on, and ignores results other than Passed, Failed and Error. Test actions fire on standard tests only, and only in projects where a Validatar admin has turned on the Standard Test Actions optional feature.

Where results land in Atlan

The custom metadata set must exist in Atlan before the first publish with Custom Metadata or Both. Atlan Custom Metadata for Validatar lists every property and shows how to create the set and its badges, by script or by hand.

Note: External DQ is a preview. Atlan does not document the structure for third-party tools, and it has not yet been seen displayed in Atlan for a tool that is not an Atlan partner. Use Custom Metadata or Both until it has been confirmed on your tenant.

Choosing what to publish

Publishing is controlled in three places, from broad to narrow, with a per-table opt-out:

1. Publish Scope. Trust Score publishes trust, quality and coverage scores per table, copied from Validatar's stored trust score. Trust + Dimensions adds one score per quality dimension, worked out by the script from each test's latest result and limited by Publish Dimensions if set. Trust + Dimensions + Tests adds individual test results. How the publish script calculates scores in the Atlan Integration article gives the calculation for each value.
2. Publish Tests. With Opted-in tests, only tests whose Publish to Atlan field is Yes are published. With All tests, every test on a matched table is published unless its field is No.
3. Test-level fields. Four project custom fields on the test, in a section named Atlan Publishing:

The fields come as their own marketplace item, Atlan Project Custom Fields. Import it once before you publish tests. To create the fields by hand instead, use exactly these reference keys, as described in Configuring Custom Fields.

Table-level opt-out. Set a table's Publish Scores to Atlan field to No to keep it out of Atlan, for example a staging table that is tested but not meant for catalog users. The field is part of the Atlan Catalog Custom Fields set.

How tables are matched. Each Validatar data source is paired with an Atlan connection, as described under Pairing data sources with Atlan connections. Within a pair, a Validatar table is matched first by its Atlan Qualified Name field, which the metadata script fills, and otherwise by schema and table name among Atlan's tables, views and materialised views. Tables found on one side only are reported as unmatched.

One Atlan connection can crawl several databases, so the same schema and table name can exist twice, such as PUBLIC.ORDERS in SALES_DB and in HR_DB. A Validatar schema named DATABASE.SCHEMA matches only that database. A bare schema name matches the Atlan table in the Validatar data source's own database. When the data source's database does not settle it, the table is reported as unmatched and the row names the databases, rather than publishing to the wrong table. Several Validatar data sources, one per database, can be paired with the same Atlan connection, and a test action publishes to the data source that holds the table's database.

Publishing part of a data source

The Publish to Atlan macro has four optional parameters. Each publish test sets its own, so one test can publish the finance schemas after the nightly load while another publishes a handful of critical tables every hour.

How the values are read:

• Entries are separated by commas, semicolons or new lines.
• * matches any run of characters, ? matches one character, and [PQ] matches P or Q.
• Case is ignored, and quotes or brackets around a name are dropped, so "public".[orders] is the same as PUBLIC.ORDERS.
• A blank parameter is no limit. In Schemas and Tables, * and all are also no limit.
• A table is published when its schema matches an entry in Schemas, it matches an entry in Tables, and it matches nothing in Exclude. A blank parameter passes every table, and Exclude wins over the other two.
• When Validatar names schemas DATABASE.SCHEMA, a schema entry matches the full name or the part after the dot. PUBLIC matches SALES_DB.PUBLIC and HR_DB.PUBLIC; SALES_DB.PUBLIC.ORD* in Tables matches only the first.

Tables outside the scope return no row; the summary line counts them as out of scope. Schemas that cannot hold a chosen table are never read, so a narrow publish stays quick on a large data source. The output names any entry that matched nothing, such as Tables pattern 'FACT_SALE' matched no table. When the parameters leave no table at all, the test returns no rows, so a test set to pass on at least one row fails instead of passing on an empty publish.

The table-level Publish Scores to Atlan field still applies inside the scope. A publish test created before template version 1.1 has no values for Schemas, Tables or Exclude and keeps publishing everything it did before.

Announcements

With Post Announcements on, a table gets a Warning announcement titled Validatar: <n> critical test(s) failing when a published test whose Announce Failures in Atlan field is Yes fails, and the announcement is removed when those tests pass again. The template changes only announcements whose title starts with Validatar:. When a table already carries someone else's announcement, Validatar leaves it in place and does not post its own. Announcements come from published tests, so they need the Trust + Dimensions + Tests scope.

Custom Fields

The object catalog needs none. Publishing reads two fields of the Atlan Catalog Custom Fields set on each table: Atlan Qualified Name, which the metadata script fills, and Publish Scores to Atlan, the per-table opt-out. It also reads the data source's Atlan Connection Qualified Name wherever Atlan Connection Map does not pair the data source. Tests opt in and get their Atlan names through the Atlan Publishing test fields above.

Installation

1. To connect to a real tenant, install pyatlan==11.4.* into the Python that runs Validatar scripts, as described under How It Works. Demo mode does not need it.
2. Open Marketplace in Validatar, find Atlan - Data Source Template and choose Get, then Import Directly. Validatar adds it under Settings > Data Source Templates. If a template named Atlan already exists, tick the option to overwrite it.
3. Create a data source from the template. If your Python scripts run on a Data Agent, assign the agent group. To try it first, set Demo Mode to true and leave the credentials blank apart from Atlan Base URL, which can be any URL.
4. Run Ingest Technical Metadata on the data source and open its catalog. Check the Ingestion Summary schema: every object type should say Loaded, or say why not.
5. For real use, turn Demo Mode off, fill in the Atlan credentials and ingest again.

To install from a file instead, download the template from the marketplace listing and import it under Settings > Data Source Templates > Import.

Setting up publishing

1. Create the Validatar custom metadata set in Atlan, as described in Atlan Custom Metadata for Validatar.
2. Import the Atlan Catalog Custom Fields and the Atlan Project Custom Fields.
3. On the Atlan data source, fill in Validatar Base URL and Validatar API Token, and choose the scope and target. Pair each warehouse data source with its Atlan connection: running the metadata script on it does this, or fill Atlan Connection Map.
4. Create a test on the Atlan data source whose test data set is the Publish to Atlan macro. To publish part of what is paired, fill Validatar Data Source IDs, Schemas, Tables or Exclude; leave them blank to publish everything. Set the result processing to row count and pass the test when at least one row comes back, since every table in scope produces a row. Add the test as the last step of the scheduled test job.
5. Run the job with Dry Run on and read the publish test's rows. When they show what you expect, turn Dry Run off.
6. For real-time results on critical tests, add a test action to those tests: trigger Test Ran, then a Run a Macro step on the Atlan data source for Push Test Result to Atlan, with its parameters set to {{test_url}}, {{test_name}} and {{result}}. Test actions need the Standard Test Actions optional feature.

Upgrading

Each Atlan data source, and the connection of each Custom Metadata and Lineage Ingestion source that uses the template, keeps its own copy of the template's pre-execution script. Importing a newer template version does not update those copies. After an upgrade, open each of these connections and make sure its Pre-Execution Script matches the one in Settings > Data Source Templates > Atlan, replacing it where it does not.

From 1.1.0 to 1.1.1 the replacement is recommended rather than required. A data source that keeps the 1.1.0 copy publishes as before, but only the 1.1.1 copy tells apart tables of the same schema and name in different databases of one Atlan connection.

From 1.0 to 1.1 this step is required on every Atlan data source. The 1.1 macros and ingestion script need code that only the 1.1 pre-execution script holds, and until the copy is replaced they stop with "This data source's Pre-Execution Script comes from an Atlan template older than 1.1.0". Existing publish tests need no change.

Troubleshooting

Related Articles

• Atlan Integration
• Atlan Custom Metadata for Validatar
• Atlan Instance Health Checks
• Atlan Metadata and Lineage Scripts
• Atlan Catalog Custom Fields
• Atlan Project Custom Fields
• Data Source Template Macros
• What Is a Data Source Template?
• Python Script Data Source Template