Overview
The Databricks data source template is designed for Databricks Lakehouse environments with Unity Catalog. It provides metadata ingestion scripts that query Unity Catalog system views, data type mappings for Spark SQL types, and profiling definitions compatible with Databricks SQL.
Platform: Databricks (with Unity Catalog)
Connection Category: Database / Databricks Engine
Template Category: Marketplace
What's Included
Connection Configuration
- Supports Databricks native connections
- Compatible with Databricks processing engine for in-cluster execution
- Identifier delimiters:
`/`
Data Type Mappings
Includes mappings for Spark SQL data types:
- String types:
STRING,VARCHAR,CHAR - Numeric types:
INT,BIGINT,SMALLINT,TINYINT,FLOAT,DOUBLE,DECIMAL - Date/time types:
DATE,TIMESTAMP,TIMESTAMP_NTZ - Boolean:
BOOLEAN - Binary:
BINARY - Complex types:
ARRAY,MAP,STRUCT
Metadata Ingestion
Queries Unity Catalog's INFORMATION_SCHEMA views to discover catalogs, schemas, tables, and columns.
Profiling
Includes standard profile definitions using Spark SQL compatible expressions.
Installation
Customization
Common customizations:
- Catalog scope — Filter to specific Unity Catalog catalogs
- Compute configuration — Set the appropriate warehouse or cluster via execution scripts
- Delta-specific profiles — Add profiles that leverage Delta Lake metadata (version history, file statistics)