Windows Directory Data Source Template

Overview

The Windows Directory data source template turns a local or network directory tree into a Validatar data source. It walks the folder tree, catalogs every file as a table, reads and profiles the common data formats (CSV, Excel, Parquet, Avro, JSON, XML, GeoJSON), and captures statistics for non-tabular files (PDF, Markdown, images, plain text). Folders become schemas, files become tables, and columns are inferred from each file's contents.

This is a Python (Script) template. All discovery and reading is done by Python running on the Data Agent, so the same template works whether the data lives on the agent's local disk or on a UNC share the agent can reach.

Platform: Windows file system (local paths and UNC shares)
Connection Category: Script (PythonScript)
Runs on: the Data Agent (required — it reads the local/UNC file system)
Template version: 2.1.0
Marketplace: Windows Directory

Note: The Windows Directory, AWS S3, and Azure Blob Storage templates are built from one shared file-reading library. They expose the same macros, profiling, and custom fields — only the storage layer (local disk vs. S3 vs. Blob) and the connection parameters differ. Learn one and you know all three.

How It Works

• Folders are schemas, files are tables. The template walks from the root folder you configure. Each folder (relative to the root, .-rooted — ., .\CSV, .\CSV\CUSTOMER_1001) becomes a schema; each file inside it becomes a table.
• Columns are inferred on read. For tabular formats, the template reads the file and infers column names and data types from the content (Parquet and Avro carry explicit types; CSV and Excel are inferred).
• One ingestion pass. Metadata ingestion runs at the data-source level and catalogs the whole tree in a single pass.
• Everything is driven by two parameters — the root path and whether to descend into subfolders.

Authentication

The Windows Directory template has no credential parameters. Access is governed entirely by the Windows identity the Data Agent runs as (its service account). There is nothing to enter in Validatar beyond the path.

Tip: If ingestion returns an empty catalog or "access denied," it is almost always the agent's service account lacking read permission on the path — not a Validatar configuration issue. Confirm the account can open the path in Windows Explorer while logged in as that identity.

Connection Parameters

Required Python Packages

These must be present on the Data Agent's Python runtime.

Required

• pandas, numpy — the core data-frame and numeric layer (hard requirement)
• pyarrow — Parquet / ORC / Feather
• fastavro — Avro
• openpyxl — Excel (.xlsx / .xlsm)

Optional (features degrade gracefully if absent)

• fitz (PyMuPDF) — PDF statistics
• Pillow (PIL) — image statistics
• charset_normalizer — best-effort text-encoding detection
• lxml — XML reading

Note: The library degrades per file, not per run. If an optional package is missing, only the files that need it are affected — that file returns a clear error and the rest of the catalog continues. Missing fastavro, for example, doesn't stop your CSVs from being read and profiled.

Capabilities

Supported File Types

Note: JSON is handled flexibly — arrays of objects, a single object, an object of arrays, GeoJSON feature collections, and line-delimited JSON (even when saved with a .json extension) are all detected and flattened into columns.

Macros

The template ships with 28 macros: 21 generic file macros and seven for nested JSON. Every generic macro works on every file type through the shared reader, so you write a test once and point it at any folder and file. Macros take metadata-linked parameters (Folder → Schema, File → Table, Column) so they render as catalog-driven dropdowns when you build a test.

A. Single-file data

B. Pattern & union (multi-file)

C. Folder & file information

D. Content statistics (including non-tabular)

Tip: Folder - Union Files Matching a Pattern is the workhorse for partitioned or daily extracts — point it at a folder of *.parquet or *.avro files and it returns one unified dataset with a _source_file column, so you can test the combined set and still trace any row back to its file.

E. Nested JSON

The macros above treat JSON as a flat table. These seven treat it as a tree.

The difference matters more than it sounds. The standard reader normalizes nested objects into dotted columns, so customer.address.city works as a column. A nested array cannot become a column, so it stays in the cell as a raw list:

order_id     customer.address.city   items
ORD-00001    Austin                  [{'sku': 'WID-001', 'quantity': 3, ...}, {...}]

You cannot count that, group it, join on it, or assert anything about it. For order payloads, event streams and API captures, where the interesting data is inside a repeated element, that is where testing usually stops.

Start with Discover Nested Structure. It is the map. Run it once against a sample file in the folder and it tells you every path, how deep it is, and which ones are arrays, so you know what to feed the other six.

Then reach for Explode an Array Path, not Flatten to Table. Both produce tabular output, but flattening with arrays=explode expands every array at once, and sibling arrays multiply: a record with 3 line items and 2 contacts yields 6 rows, and any sum over it is wrong by a factor of 2. Exploding one path does not have that problem. Flatten's default is therefore join, which keeps one row per record and encodes arrays as text.

Note: explode is the right choice when you want the cross-product, or when only one array is present. The parameter exists so the behaviour is a decision rather than a surprise.

The same 28 macros ship in the AWS S3 and Azure Blob Storage templates, so a test written against a local folder runs unchanged against a bucket or a container.

See Macros for how macros are used when building tests.

Metadata Ingestion

Ingestion runs once at the data-source level. It walks the tree, registers each folder as a schema and each file as a table, and populates the File Info catalog custom fields (see below) — including per-folder rollups (file count, total size, distinct file types, newest/oldest modified). Column-level metadata is discovered on demand by the profile set rather than during ingestion, which keeps ingestion fast on large trees.

See Metadata Ingestion Scripts — Python Templates for how Python ingestion scripts work.

Profiling

The template ships one profile: File Profile (columns + table metrics, any file type). It runs per table (file) and is type-safe — it reads any supported file and emits only the metrics that apply to that file's type. Run it as a profile set against a folder to profile every file in it.

Following the principle that anything numeric or statistical about an asset is a profile, and descriptors/tags/text belong in custom fields, it produces:

Column-level profiles (numeric, per column)

• Null count / percent, blank count / percent, distinct count / percent
• Length statistics: min / max / mean length
• Most common value and its count
• Numeric statistics (numeric columns): min, max, mean, median, standard deviation, zero count, negative count
• Date range (date columns): min date, max date

Table-level file metrics (numeric, per file)

• File size (bytes), file age (days), row count, column count
• Type-specific: PDF page count, content word/line/record count, image width/height (px)

Note: The column profiler is deliberately type-safe: a mixed-type or all-text column never breaks the run — non-applicable metrics are simply omitted for that column.

See Profiling Configuration — Python Templates for profile-set mechanics.

Catalog Custom Fields (required dependency)

The ingestion script emits catalog custom fields to describe files and folders. These fields hold descriptors — text, tags, dates, and folder rollups — that don't fit the numeric profile model (schemas in particular can't be profiled, so folder rollups are captured here).

You must import the File Info - Catalog Custom Fields set before you run ingestion. Validatar silently drops custom-field columns that have no matching field definition, so without it these values are simply lost.

Schema level (folder rollups): Contains Files, Schema Object Represents, Full Path, File Count, Total Size (bytes), Distinct File Types, Readable File Count, Newest Modified, Oldest Modified

Table level (per file): File Type, File Name, Sheet Name, File Size, File Total Bytes, File Extension, Date Modified, File Logical Table

Warning: Catalog custom fields import through the UI only (Settings > Catalog Custom Fields > Import), and require the GlobalCustomMetadataAdmin role. Import the File Info set first, then import this template, then create the data source and run ingestion.

Installation

1. Import the custom fields first. In Settings > Catalog Custom Fields > Import, import the File Info - Catalog Custom Fields set.
2. Import the template. In Settings > Data Source Templates > Import, select the Windows Directory template file.
3. Create a data source from the template and set Parent Folder Path and Include Sub Folders.
4. Assign a Data Agent that can reach the path and has the required Python packages.
5. Run metadata ingestion, then run the File Profile profile set against the folders you care about.

Upgrading

Importing a newer version of the template with Overwrite existing updates the template itself and nothing else. Each data source built from a Script template keeps its own copy of the template's pre-execution script, taken at the moment the template was selected. A data source created on 2.0.0 keeps running the 2.0.0 code until you refresh that copy.

To pick up a new version on an existing data source:

1. Import the new template file (Settings > Data Source Templates > Import, with overwrite).
2. Open each data source built from the template and edit its connection.
3. Re-select Windows Directory as the template, confirm the parameters are still filled, and save.
4. Run JSON - Discover Nested Structure against any JSON file in the tree. On a data source still carrying the old pre-execution script it fails with a NameError for the JSON helpers; on a refreshed one it returns the path listing.

New data sources created after the import pick up the current version with no extra step.

What You Still Need to Configure

• Data Agent selection — the source only works through an agent with local/UNC access to the path.
• Service-account permissions — read access on the path (and the share, for UNC).
• Python packages — confirm the required set is installed on the agent; add optional ones for PDF/image/encoding support.
• Folder scope — set include_sub_folders to false to keep the catalog shallow on very large trees.

Troubleshooting

Sample Test Pack

Windows Directory - Unstructured File Sample Tests (marketplace) ships 190 worked examples against this template: text, log and markdown files, PDFs and images, each with its adjustable values in a block at the top of the script and a switch that points it at a file built to make it fail. See Windows Directory Unstructured File Sample Tests. The files those tests read are a separate download, Unstructured File Sample Fixture (S3 and Azure Blob); unzip it and copy the six top-level folders under the data source's Parent Folder Path on the Data Agent host.

Related Articles

• What Is a Data Source Template?
• Windows Directory Unstructured File Sample Tests
• AWS S3 Data Source Template
• Azure Blob Storage Data Source Template
• Python Script Data Source Template
• Macros
• Metadata Ingestion Scripts — Python Templates
• Profiling Configuration — Python Templates