---
title: "Azure Blob Storage Data Source Template"
slug: "azure-blob-storage-data-source-template"
updated: 2026-10-06T17:13:16Z
published: 2026-10-06T17:13:16Z
canonical: "docs.validatar.com/azure-blob-storage-data-source-template"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.validatar.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Azure Blob Storage Data Source Template

## Overview

The Azure Blob Storage data source template turns a blob container (optionally scoped to a prefix) into a Validatar data source. It lists blobs with `azure-storage-blob`, catalogs every blob as a table, reads and profiles the common data formats (CSV, Excel, Parquet, Avro, JSON, XML, GeoJSON), and captures statistics for non-tabular blobs (PDF, Markdown, images, plain text). Blob prefixes become schemas, blobs become tables, and columns are inferred from each blob's contents.

This is a **Python (Script) template** that uses the same shared file-reading library as the [Windows Directory](/docs/windows-directory-data-source-template) and [AWS S3](/docs/aws-s3-data-source-template) templates. The macros, profiling, and custom fields are identical — only the storage layer (`azure-storage-blob`) and the connection parameters differ.

**Platform:** Azure Blob Storage / ADLS Gen2 **Connection Category:** Script (`PythonScript`) **Runs on:** any runtime that has `azure-storage-blob` and network reachability to the storage account — typically the Data Agent **Template version:** 1.1.0 **Marketplace:** [Azure Blob Storage](https://admin.cloud.validatar.com/marketplace/63)

> **What's new in 1.1.0:** seven macros for nested JSON (see [Nested JSON](/docs/azure-blob-storage-data-source-template#macros)); folder listings scoped to the folder, where earlier versions listed by string prefix and a prefix of `logs` also returned every blob under `logs-failing/`; a fix for pretty-printed `.json` arrays, which were misdetected as line-delimited JSON and failed to read; and a fix for folder names whose first character formed a Python escape sequence. Existing data sources need the template re-selected before any of this takes effect; see [Upgrading](/docs/azure-blob-storage-data-source-template#upgrading).

## How It Works

- **Prefixes are schemas, blobs are tables.** The template lists the container under the configured prefix. Each virtual directory becomes a schema (`.`-rooted, e.g. `.\csv\customer_1001`); each blob becomes a table.
- **Columns are inferred on read.** Blobs are read into data frames and columns/types inferred (Parquet and Avro carry explicit types; CSV and Excel are inferred).
- **One ingestion pass** catalogs the whole scope at the data-source level.
- **Scope with `container` + `prefix`** and control depth with `include_sub_folders`.

## Authentication

The template supports three authentication styles from a single template. Which one is used is determined by which credential parameters you fill in.

| Method | Parameters to fill | When to use |
| --- | --- | --- |
| **Connection string** | **Connection String** | Simplest setup; the connection string embeds the account and its key |
| **SAS token** | **Account URL** + **SAS Token** | Least-privilege, time-bounded access scoped to a container or prefix — the recommended production pattern |
| **Account key** | **Account URL** + **Account Key** | Full-account access via the storage account key, without embedding it in a connection string |

The adapter picks the method by inspecting which parameters are present: a connection string is used directly; otherwise it builds a client from the Account URL plus whichever credential (SAS token or account key) you supplied.

> **Tip:** Prefer a **SAS token** in production. Scope it to read + list on just the container (or a specific prefix) and give it an expiry, so Validatar's access is bounded in both scope and time. A SAS with only read/list permission is all this template needs.

> **Warning:** Connection strings and account keys grant broad access to the storage account. Treat them as high-value secrets, and rotate them on the account's key-rotation schedule.

## Connection Parameters

| Parameter | Reference key | Type | Required | Description |
| --- | --- | --- | --- | --- |
| Container | `container` | String | Yes | Blob container name. |
| Prefix | `prefix` | String | No | Blob-name prefix to scope the catalog. |
| Account URL | `account_url` | String | No | `https://<account>.blob.core.windows.net` (for SAS or account-key auth). |
| SAS Token | `sas_token` | Secret | No | SAS token (used with Account URL). |
| Connection String | `connection_string` | Secret | No | Full connection string (alternative to Account URL + credential). |
| Account Key | `account_key` | Secret | No | Storage account key (used with Account URL). |
| Include Sub Folders | `include_sub_folders` | Boolean | Yes | Descend into nested blob prefixes. Default: `true`. |

## Required Python Packages

These must be present on the runtime that executes the source (the Data Agent, or a cloud runtime that carries the SDK).

**Required**

- `azure-storage-blob` — the Azure Blob client (**required for this template**)
- `pandas`, `numpy` — the core data-frame and numeric layer
- `pyarrow` — Parquet / ORC / Feather
- `fastavro` — Avro
- `openpyxl` — Excel (`.xlsx` / `.xlsm`)

**Optional (features degrade gracefully if absent)**

- `fitz` (PyMuPDF) — PDF statistics
- `Pillow` (`PIL`) — image statistics
- `charset_normalizer` — text-encoding detection
- `lxml` — XML reading

> **Note:** The library degrades per blob, not per run — a missing optional package only affects the blobs that need it; the rest of the catalog continues.

## Capabilities

### Supported File Types

| Catalog file type | Extensions | Read as |
| --- | --- | --- |
| Delimited | `.csv`, `.tsv`, `.tab` | Tabular (delimiter sniffed) |
| Fixed Width / Text | `.txt`, `.flat`, `.dat` | Tabular |
| Excel | `.xlsx`, `.xlsm`, `.xls` | Tabular |
| Columnar | `.parquet`, `.orc`, `.feather` | Tabular (explicit types) |
| Row Binary | `.avro` | Tabular (explicit types) |
| Semi-Structured | `.json`, `.jsonl`, `.ndjson`, `.xml`, `.geojson` | Tabular (flattened / normalized) |
| Document | `.pdf`, `.md`, `.markdown`, `.rst` | Content statistics |
| Image | `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.tif`, `.tiff`, `.webp` | Content statistics |

> **Note:** Line-delimited JSON is detected even when a blob has a `.json` extension, and flattened into columns.

### Macros

The template ships with 28 macros: 21 generic file macros and seven for nested JSON. Because every macro goes through the shared reader, the same test works against any blob and any supported format. Parameters are metadata-linked (Folder → Schema, File → Table, Column) so they render as catalog-driven dropdowns.

**A. Single-blob data**

| Macro | Returns |
| --- | --- |
| File - Read Data (any type) | The whole blob as a dataset |
| File - First N Rows | The first N rows (default 100) |
| File - Row Count | Row count as a single value |
| File - Schema (columns + inferred types) | Column list with inferred type and null % |
| File - Profile One Column | Profile statistics for one column |
| File - Row Count Grouped by Column | Group-by counts |
| File - Aggregate a Column | `sum` / `min` / `max` / `mean` / `count` / `nunique` |

**B. Pattern & union (multi-blob)**

| Macro | Returns |
| --- | --- |
| Folder - Files Matching a Pattern | Blobs whose name matches a glob (e.g. `account_*.avro`) |
| Folder - Union Files Matching a Pattern | Every matching blob read and unioned (schema-aligned, source-tagged) |
| Folder - Row Count Across Matching Files | Per-blob and total row counts |
| Folder - Schema Consistency Across Files | Each blob's columns and whether they match the first (partitioned extracts) |

**C. Prefix & blob information**

| Macro | Returns |
| --- | --- |
| Folder - File Inventory | Every blob with size, detected type, and modified date |
| Folder - Summary | One-row rollup: blob count, total size, distinct types, date range |
| File - Full Metadata | One blob's size, modified date, MD5, and (if tabular) row/column counts |

**D. Content statistics (including non-tabular)**

| Macro | Returns |
| --- | --- |
| File - Content Stats (any type) | Type-aware stats for any blob |
| PDF - Statistics | Pages, words, embedded images, title/author, encryption flag |
| Markdown - Statistics | Headings, links, images, code blocks, table rows |
| Text - Statistics | Lines, words, encoding, blank lines, longest line |
| Image - Statistics | Width/height, megapixels, format, mode, alpha |
| JSON - Statistics | Record count, nesting depth, top-level keys |
| XML - Statistics | Root tag, element count, distinct tags, depth |

> **Tip:** **Folder - Union Files Matching a Pattern** unifies partitioned blob layouts — point it at a prefix of `*.avro` or `*.parquet` blobs and it returns one dataset with a `_source_file` column, so you can validate the combined set and still trace any row to its blob.

**E. Nested JSON**

The macros above deal with JSON as a *flat* table. These seven deal with it as a *tree*.

The distinction matters more than it sounds. The standard reader normalizes nested **objects** into dotted columns, so `customer.address.city` works as a column. A nested **array** cannot become a column, so it stays in the cell as a raw list:

```
order_id     customer.address.city   items
ORD-00001    Austin                  [{'sku': 'WID-001', 'quantity': 3, ...}, {...}]
```

You cannot count that, group it, join on it, or assert anything about it. For order payloads, event streams and API captures landed as blobs, the interesting data is inside a repeated element, and that is where testing usually stops.

| Macro | Returns |
| --- | --- |
| JSON - Discover Nested Structure | One row per path in the document tree: depth, JSON type(s), how many records contain it, array cardinality, and a sample value |
| JSON - Flatten to Table | The whole document as a table, with configurable array handling (`join` / `explode` / `count`) |
| JSON - Explode an Array Path | One row per element of a chosen array, optionally carrying the parent record's fields |
| JSON - Extract Specific Paths | A chosen set of dotted paths as columns; `[]` descends into every array element |
| JSON - Array Cardinality per Record | Element count per record for a chosen array path, keyed by a business identifier |
| JSON - Type Conflicts Across Records | Paths whose JSON type varies between records |
| JSON - Key Coverage / Sparse Fields | Paths present in fewer than N% of records |

**Start with Discover Nested Structure.** It is the map. Run it once against a sample blob and it tells you every path, how deep it is, and which ones are arrays, so you know what to feed the other six.

**Then reach for Explode an Array Path rather than Flatten to Table.** Both produce tabular output, but flattening with `arrays=explode` expands *every* array at once, and sibling arrays multiply: a payload with 3 line items and 2 contacts yields 6 rows, and any sum over it is wrong by a factor of 2. Exploding one path does not have that problem. Flatten's default is therefore `join`, which keeps one row per record and encodes arrays as text.

> **Note:** `explode` is the right choice when you genuinely want the cross-product, or when only one array is present. The parameter exists so the behaviour is a decision rather than a surprise.

See [Macros](/docs/data-source-template-macros) for how macros are used when building tests. The six worked examples in the **AWS S3 - Nested JSON Sample Tests** pack run unchanged against a blob data source once the pack's data source is mapped to yours; the storage layer is the only difference.

### Metadata Ingestion

Ingestion runs once at the data-source level. It lists the container under the prefix, registers each blob prefix as a schema and each blob as a table, and populates the **File Info** catalog custom fields — including per-prefix rollups (blob count, total size, distinct file types, newest/oldest modified). Column-level metadata is discovered on demand by the profile set.

See [Metadata Ingestion Scripts — Python Templates](/docs/metadata-ingestion-scripts-python-templates).

### Profiling

The template ships one profile: **File Profile (columns + table metrics, any file type)** — the same type-safe profiler as the Windows Directory template. Run it as a profile set against a prefix to profile every blob in it. Following the principle that *numeric/statistical values are profiles and descriptors are custom fields*, it produces:

- **Column-level profiles:** null/blank/distinct counts and percentages, length statistics, most-common value, numeric statistics (min/max/mean/median/standard deviation, zero/negative counts), and date range.
- **Table-level file metrics:** file size (bytes), file age (days), row count, column count, and type-specific metrics (PDF page count, content word/line/record count, image width/height).

See [Profiling Configuration — Python Templates](/docs/profiling-configuration-python-templates).

### Catalog Custom Fields (required dependency)

The ingestion script emits **File Info** catalog custom fields to describe blobs and prefixes — the text/tag/date descriptors and prefix rollups that don't fit the numeric profile model. Import the **File Info - Catalog Custom Fields** set **before** running ingestion; Validatar silently drops custom-field columns with no matching definition.

**Schema level (prefix rollups):** Contains Files, Schema Object Represents, Full Path, File Count, Total Size (bytes), Distinct File Types, Readable File Count, Newest Modified, Oldest Modified

**Table level (per blob):** File Type, File Name, Sheet Name, File Size, File Total Bytes, File Extension, Date Modified, File Logical Table

> **Warning:** Catalog custom fields import through the UI only (**Settings > Catalog Custom Fields > Import**) and require the **GlobalCustomMetadataAdmin** role.

## Installation

1. **Import the custom fields first** — the **File Info - Catalog Custom Fields** set (**Settings > Catalog Custom Fields > Import**).
2. **Import the template** — **Settings > Data Source Templates > Import**, select the Azure Blob Storage template file.
3. **Create a data source** and set **Container** and **Prefix**, then supply credentials for one auth method (connection string; or Account URL + SAS token; or Account URL + account key).
4. **Assign a runtime** with `azure-storage-blob` and network access to the storage account.
5. **Run metadata ingestion**, then run the **File Profile** profile set.

## Upgrading

Importing a newer version of the template with **Overwrite existing** updates the template itself and nothing else. Each data source built from a Script template keeps its own copy of the template's pre-execution script, taken at the moment the template was selected. A data source created on 1.0.0 keeps running the 1.0.0 code, without the JSON macros and with the old folder listing, until you refresh that copy.

To pick up a new version on an existing data source:

1. Import the new template file (**Settings > Data Source Templates > Import**, with overwrite).
2. Open each data source built from the template and edit its connection.
3. Re-select **Azure Blob Storage** as the template, confirm the parameters are still filled, and save.
4. Run a folder macro (for example **Folder - File Inventory** on a prefix that has a similarly named neighbour) to confirm the new behaviour.

New data sources created after the import pick up the current version with no extra step.

## What You Still Need to Configure

- **Auth method** — connection string, SAS token, or account key. Prefer a scoped, time-bounded SAS token in production.
- **Container access** — read + list on the container (or a prefix, for a scoped SAS).
- **Runtime placement** — a Data Agent or cloud runtime that carries `azure-storage-blob`.
- **Scope** — set `prefix` to avoid cataloging an entire large container, and `include_sub_folders` to control depth.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| Authentication / authorization error | Wrong or expired credential; SAS lacks read/list | Re-issue the SAS with read + list; verify the connection string or account key |
| Empty catalog | Wrong container/prefix | Verify the container name and prefix |
| No auth method resolves | None of the credential combinations are complete | Provide a connection string, or Account URL + SAS token, or Account URL + account key |
| A single blob errors | Missing optional package for that format | Install the package (`fastavro`, `openpyxl`, `pyarrow`, `fitz`, `Pillow`) |
| Custom-field values missing | File Info custom fields not imported before ingestion | Import the File Info set, then re-run ingestion |
| A folder macro or test returns blobs from a neighbouring prefix (`logs` also lists `logs-failing/…`) | Versions before 1.1.0 listed by string prefix rather than by folder | Fixed in template 1.1.0; import it and re-select the template on the data source (see [Upgrading](/docs/azure-blob-storage-data-source-template#upgrading)) |
| A folder's blobs report not found even though they exist | Folder name begins with a character that forms a Python escape (`reference`, `temp`, `feeds`, `nightly`, …) | Fixed in template 1.1.0; upgrade the template |
| A pretty-printed `.json` array fails to read with `Expecting value: line 1 column 2` | Versions before 1.1.0 misdetected indented arrays as line-delimited JSON | Fixed in template 1.1.0; upgrade the template |
| The template was upgraded but the old behaviour persists | The data source still holds its own copy of the previous pre-execution script | Re-select the template on the data source (see [Upgrading](/docs/azure-blob-storage-data-source-template#upgrading)) |
| `Columnar read failed … Repetition level histogram size mismatch` | The Parquet blob was written by a newer `pyarrow` than the runtime has | Upgrade `pyarrow` on the Data Agent; the reader is shared with the [AWS S3 template](/docs/aws-s3-data-source-template), whose article explains the version skew |

## Sample Test Pack

**Azure Blob Storage - Unstructured File Sample Tests** ([marketplace](https://admin.cloud.validatar.com/marketplace/65)) ships 190 worked examples against this template: text, log and markdown files, PDFs and images, each with its adjustable values in a block at the top of the script. See [Azure Blob Storage Unstructured File Sample Tests](/docs/azure-blob-unstructured-sample-tests). The files those tests read are a separate download, [Unstructured File Sample Fixture (S3 and Azure Blob)](https://admin.cloud.validatar.com/marketplace/66).

## Related Articles

- [What Is a Data Source Template?](/docs/what-is-a-data-source-template-overview)
- [Azure Blob Storage Unstructured File Sample Tests](/docs/azure-blob-unstructured-sample-tests)
- [Windows Directory Data Source Template](/docs/windows-directory-data-source-template)
- [AWS S3 Data Source Template](/docs/aws-s3-data-source-template)
- [Python Script Data Source Template](/docs/python-script-data-source-template)
- [Macros](/docs/data-source-template-macros)
- [Metadata Ingestion Scripts — Python Templates](/docs/metadata-ingestion-scripts-python-templates)
- [Profiling Configuration — Python Templates](/docs/profiling-configuration-python-templates)