Collibra Metadata and Lineage Scripts

Overview

Two Python scripts bring what Collibra knows about your warehouse into Validatar, and a third prepares Collibra for the results Validatar publishes:

• collibra_custom_metadata.py fills the Collibra section of the Validatar catalog on one warehouse data source: asset ids and links, descriptions, statuses, owners and stewards, tags, business terms, data categories, security classification, personal data flags, accepted data classes and any Collibra attribute you name, at data source, schema, table and, optionally, column level.
• collibra_lineage.py loads Collibra's column lineage between the databases you list as Validatar lineage, at table or column level, so a failed test shows what sits downstream.
• collibra_setup.py creates the Validatar Data Quality domain in Collibra and checks the attributes, relation types and dimensions Validatar's rules need. A Collibra admin runs it once. Collibra Setup for Validatar covers it in full.

They are part of the Collibra Integration. Publishing Validatar results to Collibra is done by the Collibra Data Source Template, not by these scripts.

Marketplace: Collibra Metadata and Lineage Scripts (a download; it is not imported)

What is in the download

How the pieces fit

Every Collibra connection in Validatar uses the Collibra data source template with the same settings: the same Collibra URL and the same credentials. What differs from one warehouse to the next sits at the top of the script, in plain Python:

# ---- 1. WHICH COLLIBRA DATABASE ----
COLLIBRA_DATABASE_FULL_NAME = "Snowflake-Connection>SAMPLE_DW"
ONLY_SCHEMAS = []                # e.g. ["PUBLIC", "FINANCE"]; empty means all

A database registered in Collibra through Edge gives every asset under it a full name that starts with the Edge connection's name and the database: Snowflake-Connection>SAMPLE_DW>STAR_SCHEMA>FACT_SALES is a table, and a column adds >CUSTOMER_KEY(column). The scripts find your tables by that prefix. The Collibra data source's own catalog in Validatar lists every database with its full name, in schema Database.

Below section 1, section 2 switches each field on or off, and a block describes the dataframes the script returns, with an example row. The template's pre-execution script carries the shared code: the Collibra client, the Validatar client and name matching. The client uses the Python standard library only, so there is nothing to install, and it reads in bulk through Collibra's Knowledge Graph GraphQL API.

To connect another warehouse, copy the script into that data source's Custom Metadata source and change section 1. Each process has its own connection, so enter the same Collibra settings on each.

Before you start

• From the marketplace, import the Collibra Data Source Template and the Collibra Catalog Custom Fields. The fields must exist before the first run: Validatar ignores a script column that has no field with the same reference key.
• In Collibra: Collibra Platform 2024.02 or later, and an Integration OAuth application, or a user name and password for Basic sign-in. Its user needs read access to the physical data dictionary domains of your databases.
• In Validatar: Govern Edition for catalog custom fields; for lineage, the Data Lineage feature and the Global Lineage Metadata Admin role.
• Run technical metadata ingestion on your warehouse data sources first. The scripts only write to schemas, tables and columns that Validatar has cataloged.

What the metadata script brings back

People come back as text, such as Dana Steward <dana@acme.example>, so a name Validatar does not know as a user never fails the run. Tags, terms, categories and data classes come back as lists. Statuses and security classifications are text, because every Collibra instance defines its own values.

The table field Collibra Asset ID is how the Collibra data source template finds the table when it publishes Validatar's results, so run this script before the first publish.

Section 2 of the script controls what comes back:

A Dropdown field rejects the whole run when it receives a value that is not one of its options, so bring Collibra attributes into Short text fields unless you know every value Collibra can send.

Set up the metadata script

1. Open the warehouse data source and go to Custom Metadata > Metadata Sources > New Source > New Query Source.
2. Untick Same as primary and Add a connection: Python Script, template Collibra, the agent group if your Python runs on a Data Agent, and your Collibra settings. The Validatar URL and token are not needed for this script.
3. Save, then Configure Ingestion Script: paste collibra_custom_metadata.py, set section 1, and tick Ignore Missing Metadata Objects so Collibra assets Validatar has not cataloged are skipped instead of failing the run.
4. Preview Script shows the rows for each level before anything is written.
5. Update Ingestion Script, Save, Refresh Now, then Set Refresh Schedule (usually daily). Metadata Ingestions lists what was added, modified and skipped at each level.

Note: Validatar gives each data source one custom metadata connection, shared by all its metadata sources. If the data source already has one, for example for the Alation or Atlan integration, step 2 shows that connection with Edit in place of Add. Editing it changes the connection for the existing sources too, so one data source takes its catalog metadata from one catalog.

Set up the lineage script

In section 1, list each Collibra database with the Validatar data source for the same warehouse:

DATA_SOURCES = [
    ("Snowflake-Connection>SAMPLE_DW", 12),   # (Collibra database full name, Validatar data source id)
]
LINEAGE_LEVEL = "table"     # or "column"

By default the script reads business lineage: the "Data Element targets Data Element" relations that Collibra Data Lineage writes between the columns it has stitched to catalog assets. Collibra keeps no table-level lineage relation, so table lineage is rolled up from those column edges. Lineage between two listed databases is included; an edge whose other end is in a database you did not list is left out.

1. Go to Settings > Lineage Metadata > Sources > New Source > New Query Source.
2. Add a connection: Python Script, template Collibra, the same Collibra settings, plus Validatar Base URL and Validatar API Token. The script uses the token to read the catalog, so it only sends edges whose two ends Validatar has cataloged: one unknown table fails a whole lineage batch.
3. Configure Ingestion SQL: paste collibra_lineage.py. Preview Script shows the edges.
4. Update Ingestion SQL, Save, then Run Import. Wait on the page while it validates the query. Validatar then asks which data source each server and database pair belongs to. When more than one Validatar data source points at the same warehouse and database, it suggests one of them, so check that the choice matches the Validatar id in DATA_SOURCES before you choose Run.
5. Set Refresh Schedule (usually weekly).

The run message counts the edges left out. Each run replaces the sources of every target table it sends, so hand-edited lineage on those tables does not survive the next run.

Run from the command line

Use this to see the dataframes before setting anything up in Validatar, or to run the scripts from your own scheduler. Keep the .py files in one folder, install requirements.txt, and set COLLIBRA_BASE_URL, COLLIBRA_AUTH_METHOD (OAuth Client or Basic), the OAuth client id and secret or the user name and password, VALIDATAR_BASE_URL and VALIDATAR_API_TOKEN.

Try it without Collibra

Set Demo Mode to Yes on the Collibra connection in Validatar, or set COLLIBRA_DEMO_MODE=1 on the command line. The scripts then answer from a built-in sample instance, and nothing is sent to Collibra. The lineage script still reads the Validatar catalog, so it needs the Validatar URL and token even in Demo Mode.

Troubleshooting

Related

• Collibra Integration
• Collibra Catalog Custom Fields
• Collibra Data Source Template
• Collibra Setup for Validatar
• Collibra Instance Health Checks

Note: The scripts are built from Collibra's documentation and published OpenAPI specifications, and have been tested offline and against a built-in sample Collibra instance. Against that instance the metadata script returned 2 schemas and 10 tables on SAMPLE_DW, and the lineage script returned 4 table edges. Inside Validatar, run as a Custom Metadata source on a Snowflake data source, the metadata script stored 6 data source, 12 schema, 102 table and 78 column values, skipping the one table Validatar had not cataloged; run as a Lineage Ingestion source, the lineage script loaded 4 edges. They have not yet run against a live Collibra instance, so expect small adjustments after your first live run.