Azure Blob Storage Data Source Template

Overview

The Azure Blob Storage data source template turns a blob container (optionally scoped to a prefix) into a Validatar data source. It lists blobs with azure-storage-blob, catalogs every blob as a table, reads and profiles the common data formats (CSV, Excel, Parquet, Avro, JSON, XML, GeoJSON), and captures statistics for non-tabular blobs (PDF, Markdown, images, plain text). Blob prefixes become schemas, blobs become tables, and columns are inferred from each blob's contents.

This is a Python (Script) template that uses the same shared file-reading library as the Windows Directory and AWS S3 templates. The macros, profiling, and custom fields are identical — only the storage layer (azure-storage-blob) and the connection parameters differ.

Platform: Azure Blob Storage / ADLS Gen2
Connection Category: Script (PythonScript)
Runs on: any runtime that has azure-storage-blob and network reachability to the storage account — typically the Data Agent
Template version: 1.1.0
Marketplace: Azure Blob Storage

What's new in 1.1.0: seven macros for nested JSON (see Nested JSON); folder listings scoped to the folder, where earlier versions listed by string prefix and a prefix of logs also returned every blob under logs-failing/; a fix for pretty-printed .json arrays, which were misdetected as line-delimited JSON and failed to read; and a fix for folder names whose first character formed a Python escape sequence. Existing data sources need the template re-selected before any of this takes effect; see Upgrading.

How It Works

• Prefixes are schemas, blobs are tables. The template lists the container under the configured prefix. Each virtual directory becomes a schema (.-rooted, e.g. .\csv\customer_1001); each blob becomes a table.
• Columns are inferred on read. Blobs are read into data frames and columns/types inferred (Parquet and Avro carry explicit types; CSV and Excel are inferred).
• One ingestion pass catalogs the whole scope at the data-source level.
• Scope with container + prefix and control depth with include_sub_folders.

Authentication

The template supports three authentication styles from a single template. Which one is used is determined by which credential parameters you fill in.

The adapter picks the method by inspecting which parameters are present: a connection string is used directly; otherwise it builds a client from the Account URL plus whichever credential (SAS token or account key) you supplied.

Tip: Prefer a SAS token in production. Scope it to read + list on just the container (or a specific prefix) and give it an expiry, so Validatar's access is bounded in both scope and time. A SAS with only read/list permission is all this template needs.

Warning: Connection strings and account keys grant broad access to the storage account. Treat them as high-value secrets, and rotate them on the account's key-rotation schedule.

Connection Parameters

Required Python Packages

These must be present on the runtime that executes the source (the Data Agent, or a cloud runtime that carries the SDK).

Required

• azure-storage-blob — the Azure Blob client (required for this template)
• pandas, numpy — the core data-frame and numeric layer
• pyarrow — Parquet / ORC / Feather
• fastavro — Avro
• openpyxl — Excel (.xlsx / .xlsm)

Optional (features degrade gracefully if absent)

• fitz (PyMuPDF) — PDF statistics
• Pillow (PIL) — image statistics
• charset_normalizer — text-encoding detection
• lxml — XML reading

Note: The library degrades per blob, not per run — a missing optional package only affects the blobs that need it; the rest of the catalog continues.

Capabilities

Supported File Types

Note: Line-delimited JSON is detected even when a blob has a .json extension, and flattened into columns.

Macros

The template ships with 28 macros: 21 generic file macros and seven for nested JSON. Because every macro goes through the shared reader, the same test works against any blob and any supported format. Parameters are metadata-linked (Folder → Schema, File → Table, Column) so they render as catalog-driven dropdowns.

A. Single-blob data

B. Pattern & union (multi-blob)

C. Prefix & blob information

D. Content statistics (including non-tabular)

Tip: Folder - Union Files Matching a Pattern unifies partitioned blob layouts — point it at a prefix of *.avro or *.parquet blobs and it returns one dataset with a _source_file column, so you can validate the combined set and still trace any row to its blob.

E. Nested JSON

The macros above deal with JSON as a flat table. These seven deal with it as a tree.

The distinction matters more than it sounds. The standard reader normalizes nested objects into dotted columns, so customer.address.city works as a column. A nested array cannot become a column, so it stays in the cell as a raw list:

order_id     customer.address.city   items
ORD-00001    Austin                  [{'sku': 'WID-001', 'quantity': 3, ...}, {...}]

You cannot count that, group it, join on it, or assert anything about it. For order payloads, event streams and API captures landed as blobs, the interesting data is inside a repeated element, and that is where testing usually stops.

Start with Discover Nested Structure. It is the map. Run it once against a sample blob and it tells you every path, how deep it is, and which ones are arrays, so you know what to feed the other six.

Then reach for Explode an Array Path rather than Flatten to Table. Both produce tabular output, but flattening with arrays=explode expands every array at once, and sibling arrays multiply: a payload with 3 line items and 2 contacts yields 6 rows, and any sum over it is wrong by a factor of 2. Exploding one path does not have that problem. Flatten's default is therefore join, which keeps one row per record and encodes arrays as text.

Note: explode is the right choice when you genuinely want the cross-product, or when only one array is present. The parameter exists so the behaviour is a decision rather than a surprise.

See Macros for how macros are used when building tests. The six worked examples in the AWS S3 - Nested JSON Sample Tests pack run unchanged against a blob data source once the pack's data source is mapped to yours; the storage layer is the only difference.

Metadata Ingestion

Ingestion runs once at the data-source level. It lists the container under the prefix, registers each blob prefix as a schema and each blob as a table, and populates the File Info catalog custom fields — including per-prefix rollups (blob count, total size, distinct file types, newest/oldest modified). Column-level metadata is discovered on demand by the profile set.

See Metadata Ingestion Scripts — Python Templates.

Profiling

The template ships one profile: File Profile (columns + table metrics, any file type) — the same type-safe profiler as the Windows Directory template. Run it as a profile set against a prefix to profile every blob in it. Following the principle that numeric/statistical values are profiles and descriptors are custom fields, it produces:

• Column-level profiles: null/blank/distinct counts and percentages, length statistics, most-common value, numeric statistics (min/max/mean/median/standard deviation, zero/negative counts), and date range.
• Table-level file metrics: file size (bytes), file age (days), row count, column count, and type-specific metrics (PDF page count, content word/line/record count, image width/height).

See Profiling Configuration — Python Templates.

Catalog Custom Fields (required dependency)

The ingestion script emits File Info catalog custom fields to describe blobs and prefixes — the text/tag/date descriptors and prefix rollups that don't fit the numeric profile model. Import the File Info - Catalog Custom Fields set before running ingestion; Validatar silently drops custom-field columns with no matching definition.

Schema level (prefix rollups): Contains Files, Schema Object Represents, Full Path, File Count, Total Size (bytes), Distinct File Types, Readable File Count, Newest Modified, Oldest Modified

Table level (per blob): File Type, File Name, Sheet Name, File Size, File Total Bytes, File Extension, Date Modified, File Logical Table

Warning: Catalog custom fields import through the UI only (Settings > Catalog Custom Fields > Import) and require the GlobalCustomMetadataAdmin role.

Installation

1. Import the custom fields first — the File Info - Catalog Custom Fields set (Settings > Catalog Custom Fields > Import).
2. Import the template — Settings > Data Source Templates > Import, select the Azure Blob Storage template file.
3. Create a data source and set Container and Prefix, then supply credentials for one auth method (connection string; or Account URL + SAS token; or Account URL + account key).
4. Assign a runtime with azure-storage-blob and network access to the storage account.
5. Run metadata ingestion, then run the File Profile profile set.

Upgrading

Importing a newer version of the template with Overwrite existing updates the template itself and nothing else. Each data source built from a Script template keeps its own copy of the template's pre-execution script, taken at the moment the template was selected. A data source created on 1.0.0 keeps running the 1.0.0 code, without the JSON macros and with the old folder listing, until you refresh that copy.

To pick up a new version on an existing data source:

1. Import the new template file (Settings > Data Source Templates > Import, with overwrite).
2. Open each data source built from the template and edit its connection.
3. Re-select Azure Blob Storage as the template, confirm the parameters are still filled, and save.
4. Run a folder macro (for example Folder - File Inventory on a prefix that has a similarly named neighbour) to confirm the new behaviour.

New data sources created after the import pick up the current version with no extra step.

What You Still Need to Configure

• Auth method — connection string, SAS token, or account key. Prefer a scoped, time-bounded SAS token in production.
• Container access — read + list on the container (or a prefix, for a scoped SAS).
• Runtime placement — a Data Agent or cloud runtime that carries azure-storage-blob.
• Scope — set prefix to avoid cataloging an entire large container, and include_sub_folders to control depth.

Troubleshooting

Sample Test Pack

Azure Blob Storage - Unstructured File Sample Tests (marketplace) ships 190 worked examples against this template: text, log and markdown files, PDFs and images, each with its adjustable values in a block at the top of the script. See Azure Blob Storage Unstructured File Sample Tests. The files those tests read are a separate download, Unstructured File Sample Fixture (S3 and Azure Blob).

Related Articles

• What Is a Data Source Template?
• Azure Blob Storage Unstructured File Sample Tests
• Windows Directory Data Source Template
• AWS S3 Data Source Template
• Python Script Data Source Template
• Macros
• Metadata Ingestion Scripts — Python Templates
• Profiling Configuration — Python Templates