Connect Validatar to Databricks with the Databricks Driver connection type and the built-in Databricks data source template. Validatar signs in as a Databricks service principal with OAuth, runs its queries on a SQL warehouse, and catalogs one Unity Catalog catalog per data source.
Applies to: Validatar Cloud · Validatar Server · 2026.1 and later
The Databricks Driver connection type arrived in Validatar 2026.1. It talks to the Databricks SQL Statement Execution API over HTTPS, so there is no ODBC driver to install on Validatar Server or a Data Agent. When you finish this page, the catalog's schemas, tables and columns are in the Validatar Catalog, ready to profile and test.
A data source or a processing engine?
Databricks shows up in two places in Validatar:
| Databricks data source | Databricks data processing engine | |
|---|---|---|
| What it is | A connection to one Unity Catalog catalog that Validatar catalogs, profiles and tests | Where Validatar runs comparisons and stores detailed results, inside your Databricks workspace |
| Where you set it up | Settings > Data Sources | Settings > Data Processing Engines |
| Connection type | Databricks Driver, or Databricks Engine when it runs through an engine | Not a data source |
| Needed for | Every Databricks catalog you test | Optional, and licensed. See Creating a Databricks Data Processing Engine. |
Both sign in the same way, with a service principal's OAuth client ID and secret, so one service principal can serve both.
Before you start
- Validatar 2026.1 or later, and the Data Source Admin role.
- A Databricks workspace with Unity Catalog and a SQL warehouse. The template reads each catalog's
information_schema, which Unity Catalog provides. - A service principal with an OAuth secret. You need its client ID and the secret. Creating a Databricks Data Processing Engine walks through creating one, granting it workspace access and Can use on the SQL warehouse.
- Read access for the service principal on the catalog you want to test.
- The warehouse ID. Find it on the warehouse's Connection details tab, or as the last part of the warehouse URL.
Validatar Cloud reaches the workspace over the internet. On Validatar Server, or through a Data Agent, that host needs outbound HTTPS to the workspace.
A catalog-level grant covers every schema and table in it, including ones created later:
-- Read access to one catalog for the Validatar service principal.
-- Replace the catalog name and the service principal application ID.
GRANT USE CATALOG ON CATALOG `<catalog>` TO `<application-id>`;
GRANT USE SCHEMA ON CATALOG `<catalog>` TO `<application-id>`;
GRANT SELECT ON CATALOG `<catalog>` TO `<application-id>`;
Create the Databricks data source
- Go to Settings > Data Sources and click New Data Source.
- Enter a Name, choose the Environment and add a Description.
- In the Connection panel, click Add.
- Leave Processing Engine on Built-In Engine: Save results in Local Storage.
- Choose Databricks Driver as the Connection Type. Validatar selects the Databricks template.
- Leave Data Agent Group on No Assigned Group, unless your workspace accepts connections only from your own network.
- Click Build. The Build Connection String dialog opens.
- Fill in the five fields (table below) and click Update Connection String.
- Enter the service principal's OAuth secret in Password.
- Click Test Connection and wait for Connection succeeded.
- Click Create Connection, then Save.
| Field in Build | Enter | Example |
|---|---|---|
| Databricks Host Domain | The workspace host name. https:// is optional. |
adb-1234567890123456.7.azuredatabricks.net |
| Database Name | The Unity Catalog catalog to catalog and test | <catalog> |
| Warehouse Name | The SQL warehouse ID, not its display name | <warehouse-id> |
| Client ID | The service principal's application (client) ID | <application-id> |
| OAuth Scope | Leave the default, all-apis |
all-apis |
The finished connection string keeps the secret out of the text with the [[PASSWORD]] placeholder:
host=adb-1234567890123456.7.azuredatabricks.net;database=<catalog>;warehouse=<warehouse-id>;client_id=<application-id>;scope=all-apis;client_secret=[[PASSWORD]];
Authentication
The Databricks Driver connection type signs in with OAuth machine-to-machine credentials only: a service principal's client ID and secret. Personal access tokens are not accepted. When the secret expires, the data source stops connecting until you generate a new secret in Databricks and enter it in Password, so note the expiry date.
Add it to a project and ingest
- On the data source, open Projects, check the projects that will test it, and click Save.
- Open Schema Metadata and click Refresh Now.
- Wait for the run in Ingestion History to finish.
The Databricks template reads the catalog's information_schema: every schema except information_schema, tables and views with their definitions, and columns with their full data type, nullability, identity, primary key constraints and comments. To test several catalogs, create one data source per catalog. Databricks Data Source Template lists what the template maps and profiles.
Check that it worked
- Metadata Statistics on Schema Metadata shows the schema, table and column counts.
- In Catalog, select the data source. The Schemas tab lists the catalog's schemas.
- Open a table and its Preview tab. Rows come back from the SQL warehouse.
A stopped warehouse starts when the first query arrives, so the first test or ingestion after a quiet spell can take longer than the rest.
Connect through a Databricks processing engine
If your team has set up a Databricks data processing engine, a data source can use the engine's connection instead of its own:
- In the connection dialog, choose the engine in Processing Engine. The Connection Type changes to Databricks Engine.
- Choose the Database (the catalog). The list shows the catalogs the engine's service principal can read.
- Leave Schema on Use default schema, or choose one.
- Click Test Connection, Create Connection and Save.
There are no credentials to enter: the source signs in as the engine's service principal. Tests on this source run inside Databricks and keep their detailed results in the engine's results catalog.
Do it with Validatar AI
Do it with Validatar AI (Validatar Cloud, beta). Open the AI panel and ask: "Create a Databricks Driver data source named Lakehouse for the catalog
on host , warehouse ID and client ID , using the Databricks template. Leave the client secret blank." Review what it creates before you run it. You can undo the change from the chat history.
Enter the secret yourself: open the data source's Connection page, click Edit, type it in Password, test and save. Never paste secrets into the chat. The AI cannot ingest metadata, so click Refresh Now afterwards.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
invalid_client or another OAuth error on Test Connection |
Wrong client ID, wrong or expired secret, or a secret from another workspace account | Generate a new secret for the service principal and enter it in Password. Check Client ID. |
| "Missing ClientSecret" | Password is empty | Enter the OAuth secret in Password. |
| An error about the warehouse | Warehouse Name holds the warehouse display name or HTTP path, or the service principal lacks Can use | Enter the warehouse ID. Grant Can use on the warehouse. |
| Host or name resolution errors | Databricks Host Domain is wrong, or includes a path | Enter only the host name, such as adb-1234567890123456.7.azuredatabricks.net. |
| Ingestion finds no schemas or tables | Database Name is not the catalog name, or the service principal lacks USE CATALOG, USE SCHEMA or SELECT |
Fix the catalog name or run the grants above, then click Refresh Now. |
Tables in hive_metastore are missing |
The template reads Unity Catalog information_schema |
Test Unity Catalog tables, or migrate the tables to Unity Catalog. |
| The data source stopped working after months | The OAuth secret expired | Generate a new secret and update Password. |
Related
- Create a Data Source
- Databricks Data Source Template
- Creating a Databricks Data Processing Engine
- Ingesting Schema Metadata
- Create and Run a Profile Set
Engineers who script their setup can create Databricks data sources through the Validatar MCP server and the Validatar Agent Kits.