Azure Blob Storage Data Source Template

Overview

The Azure Blob Storage data source template turns a blob container (optionally scoped to a prefix) into a Validatar data source. It lists blobs with azure-storage-blob, catalogs every blob as a table, reads and profiles the common data formats (CSV, Excel, Parquet, Avro, JSON, XML, GeoJSON), and captures statistics for non-tabular blobs (PDF, Markdown, images, plain text). Blob prefixes become schemas, blobs become tables, and columns are inferred from each blob's contents.

This is a Python (Script) template that uses the same shared file-reading library as the Windows Directory and AWS S3 templates. The macros, profiling, and custom fields are identical — only the storage layer (azure-storage-blob) and the connection parameters differ.

Platform: Azure Blob Storage / ADLS Gen2
Connection Category: Script (PythonScript)
Runs on: any runtime that has azure-storage-blob and network reachability to the storage account — typically the Data Agent
Template version: 1.0.0

How It Works

• Prefixes are schemas, blobs are tables. The template lists the container under the configured prefix. Each virtual directory becomes a schema (.-rooted, e.g. .\csv\customer_1001); each blob becomes a table.
• Columns are inferred on read. Blobs are read into data frames and columns/types inferred (Parquet and Avro carry explicit types; CSV and Excel are inferred).
• One ingestion pass catalogs the whole scope at the data-source level.
• Scope with container + prefix and control depth with include_sub_folders.

Authentication

The template supports three authentication styles from a single template. Which one is used is determined by which credential parameters you fill in.

The adapter picks the method by inspecting which parameters are present: a connection string is used directly; otherwise it builds a client from the Account URL plus whichever credential (SAS token or account key) you supplied.

Tip: Prefer a SAS token in production. Scope it to read + list on just the container (or a specific prefix) and give it an expiry, so Validatar's access is bounded in both scope and time. A SAS with only read/list permission is all this template needs.

Warning: Connection strings and account keys grant broad access to the storage account. Treat them as high-value secrets, and rotate them on the account's key-rotation schedule.

Connection Parameters

Required Python Packages

These must be present on the runtime that executes the source (the Data Agent, or a cloud runtime that carries the SDK).

Required

• azure-storage-blob — the Azure Blob client (required for this template)
• pandas, numpy — the core data-frame and numeric layer
• pyarrow — Parquet / ORC / Feather
• fastavro — Avro
• openpyxl — Excel (.xlsx / .xlsm)

Optional (features degrade gracefully if absent)

• fitz (PyMuPDF) — PDF statistics
• Pillow (PIL) — image statistics
• charset_normalizer — text-encoding detection
• lxml — XML reading

Note: The library degrades per blob, not per run — a missing optional package only affects the blobs that need it; the rest of the catalog continues.

Capabilities

Supported File Types

Note: Line-delimited JSON is detected even when a blob has a .json extension, and flattened into columns.

Macros

The template ships with 21 generic macros. Because every macro goes through the shared reader, the same test works against any blob and any supported format. Parameters are metadata-linked (Folder → Schema, File → Table, Column) so they render as catalog-driven dropdowns.

A. Single-blob data

B. Pattern & union (multi-blob)

C. Prefix & blob information

D. Content statistics (including non-tabular)

Tip: Folder - Union Files Matching a Pattern unifies partitioned blob layouts — point it at a prefix of *.avro or *.parquet blobs and it returns one dataset with a _source_file column, so you can validate the combined set and still trace any row to its blob.

See Macros for how macros are used when building tests.

Metadata Ingestion

Ingestion runs once at the data-source level. It lists the container under the prefix, registers each blob prefix as a schema and each blob as a table, and populates the File Info catalog custom fields — including per-prefix rollups (blob count, total size, distinct file types, newest/oldest modified). Column-level metadata is discovered on demand by the profile set.

See Metadata Ingestion Scripts — Python Templates.

Profiling

The template ships one profile: File Profile (columns + table metrics, any file type) — the same type-safe profiler as the Windows Directory template. Run it as a profile set against a prefix to profile every blob in it. Following the principle that numeric/statistical values are profiles and descriptors are custom fields, it produces:

• Column-level profiles: null/blank/distinct counts and percentages, length statistics, most-common value, numeric statistics (min/max/mean/median/standard deviation, zero/negative counts), and date range.
• Table-level file metrics: file size (bytes), file age (days), row count, column count, and type-specific metrics (PDF page count, content word/line/record count, image width/height).

See Profiling Configuration — Python Templates.

Catalog Custom Fields (required dependency)

The ingestion script emits File Info catalog custom fields to describe blobs and prefixes — the text/tag/date descriptors and prefix rollups that don't fit the numeric profile model. Import the File Info - Catalog Custom Fields set before running ingestion; Validatar silently drops custom-field columns with no matching definition.

Schema level (prefix rollups): Contains Files, Schema Object Represents, Full Path, File Count, Total Size (bytes), Distinct File Types, Readable File Count, Newest Modified, Oldest Modified

Table level (per blob): File Type, File Name, Sheet Name, File Size, File Total Bytes, File Extension, Date Modified, File Logical Table

Warning: Catalog custom fields import through the UI only (Settings > Catalog Custom Fields > Import) and require the GlobalCustomMetadataAdmin role.

Installation

1. Import the custom fields first — the File Info - Catalog Custom Fields set (Settings > Catalog Custom Fields > Import).
2. Import the template — Settings > Data Source Templates > Import, select the Azure Blob Storage template file.
3. Create a data source and set Container and Prefix, then supply credentials for one auth method (connection string; or Account URL + SAS token; or Account URL + account key).
4. Assign a runtime with azure-storage-blob and network access to the storage account.
5. Run metadata ingestion, then run the File Profile profile set.

What You Still Need to Configure

• Auth method — connection string, SAS token, or account key. Prefer a scoped, time-bounded SAS token in production.
• Container access — read + list on the container (or a prefix, for a scoped SAS).
• Runtime placement — a Data Agent or cloud runtime that carries azure-storage-blob.
• Scope — set prefix to avoid cataloging an entire large container, and include_sub_folders to control depth.

Troubleshooting

Related Articles

• What Is a Data Source Template?
• Windows Directory Data Source Template
• AWS S3 Data Source Template
• Python Script Data Source Template
• Macros
• Metadata Ingestion Scripts — Python Templates
• Profiling Configuration — Python Templates