Windows Directory Data Source Template

Overview

The Windows Directory data source template turns a local or network directory tree into a Validatar data source. It walks the folder tree, catalogs every file as a table, reads and profiles the common data formats (CSV, Excel, Parquet, Avro, JSON, XML, GeoJSON), and captures statistics for non-tabular files (PDF, Markdown, images, plain text). Folders become schemas, files become tables, and columns are inferred from each file's contents.

This is a Python (Script) template. All discovery and reading is done by Python running on the Data Agent, so the same template works whether the data lives on the agent's local disk or on a UNC share the agent can reach.

Platform: Windows file system (local paths and UNC shares)
Connection Category: Script (PythonScript)
Runs on: the Data Agent (required — it reads the local/UNC file system)
Template version: 2.1.0

Note: The Windows Directory, AWS S3, and Azure Blob Storage templates are built from one shared file-reading library. They expose the same macros, profiling, and custom fields — only the storage layer (local disk vs. S3 vs. Blob) and the connection parameters differ. Learn one and you know all three.

How It Works

• Folders are schemas, files are tables. The template walks from the root folder you configure. Each folder (relative to the root, .-rooted — ., .\CSV, .\CSV\CUSTOMER_1001) becomes a schema; each file inside it becomes a table.
• Columns are inferred on read. For tabular formats, the template reads the file and infers column names and data types from the content (Parquet and Avro carry explicit types; CSV and Excel are inferred).
• One ingestion pass. Metadata ingestion runs at the data-source level and catalogs the whole tree in a single pass.
• Everything is driven by two parameters — the root path and whether to descend into subfolders.

Authentication

The Windows Directory template has no credential parameters. Access is governed entirely by the Windows identity the Data Agent runs as (its service account). There is nothing to enter in Validatar beyond the path.

Tip: If ingestion returns an empty catalog or "access denied," it is almost always the agent's service account lacking read permission on the path — not a Validatar configuration issue. Confirm the account can open the path in Windows Explorer while logged in as that identity.

Connection Parameters

Required Python Packages

These must be present on the Data Agent's Python runtime.

Required

• pandas, numpy — the core data-frame and numeric layer (hard requirement)
• pyarrow — Parquet / ORC / Feather
• fastavro — Avro
• openpyxl — Excel (.xlsx / .xlsm)

Optional (features degrade gracefully if absent)

• fitz (PyMuPDF) — PDF statistics
• Pillow (PIL) — image statistics
• charset_normalizer — best-effort text-encoding detection
• lxml — XML reading

Note: The library degrades per file, not per run. If an optional package is missing, only the files that need it are affected — that file returns a clear error and the rest of the catalog continues. Missing fastavro, for example, doesn't stop your CSVs from being read and profiled.

Capabilities

Supported File Types

Note: JSON is handled flexibly — arrays of objects, a single object, an object of arrays, GeoJSON feature collections, and line-delimited JSON (even when saved with a .json extension) are all detected and flattened into columns.

Macros

The template ships with 21 generic macros. Every macro works on every file type through the shared reader, so you write a test once and point it at any folder and file. Macros take metadata-linked parameters (Folder → Schema, File → Table, Column) so they render as catalog-driven dropdowns when you build a test.

A. Single-file data

B. Pattern & union (multi-file)

C. Folder & file information

D. Content statistics (including non-tabular)

Tip: Folder - Union Files Matching a Pattern is the workhorse for partitioned or daily extracts — point it at a folder of *.parquet or *.avro files and it returns one unified dataset with a _source_file column, so you can test the combined set and still trace any row back to its file.

See Macros for how macros are used when building tests.

Metadata Ingestion

Ingestion runs once at the data-source level. It walks the tree, registers each folder as a schema and each file as a table, and populates the File Info catalog custom fields (see below) — including per-folder rollups (file count, total size, distinct file types, newest/oldest modified). Column-level metadata is discovered on demand by the profile set rather than during ingestion, which keeps ingestion fast on large trees.

See Metadata Ingestion Scripts — Python Templates for how Python ingestion scripts work.

Profiling

The template ships one profile: File Profile (columns + table metrics, any file type). It runs per table (file) and is type-safe — it reads any supported file and emits only the metrics that apply to that file's type. Run it as a profile set against a folder to profile every file in it.

Following the principle that anything numeric or statistical about an asset is a profile, and descriptors/tags/text belong in custom fields, it produces:

Column-level profiles (numeric, per column)

• Null count / percent, blank count / percent, distinct count / percent
• Length statistics: min / max / mean length
• Most common value and its count
• Numeric statistics (numeric columns): min, max, mean, median, standard deviation, zero count, negative count
• Date range (date columns): min date, max date

Table-level file metrics (numeric, per file)

• File size (bytes), file age (days), row count, column count
• Type-specific: PDF page count, content word/line/record count, image width/height (px)

Note: The column profiler is deliberately type-safe: a mixed-type or all-text column never breaks the run — non-applicable metrics are simply omitted for that column.

See Profiling Configuration — Python Templates for profile-set mechanics.

Catalog Custom Fields (required dependency)

The ingestion script emits catalog custom fields to describe files and folders. These fields hold descriptors — text, tags, dates, and folder rollups — that don't fit the numeric profile model (schemas in particular can't be profiled, so folder rollups are captured here).

You must import the File Info - Catalog Custom Fields set before you run ingestion. Validatar silently drops custom-field columns that have no matching field definition, so without it these values are simply lost.

Schema level (folder rollups): Contains Files, Schema Object Represents, Full Path, File Count, Total Size (bytes), Distinct File Types, Readable File Count, Newest Modified, Oldest Modified

Table level (per file): File Type, File Name, Sheet Name, File Size, File Total Bytes, File Extension, Date Modified, File Logical Table

Warning: Catalog custom fields import through the UI only (Settings > Catalog Custom Fields > Import), and require the GlobalCustomMetadataAdmin role. Import the File Info set first, then import this template, then create the data source and run ingestion.

Installation

1. Import the custom fields first. In Settings > Catalog Custom Fields > Import, import the File Info - Catalog Custom Fields set.
2. Import the template. In Settings > Data Source Templates > Import, select the Windows Directory template file.
3. Create a data source from the template and set Parent Folder Path and Include Sub Folders.
4. Assign a Data Agent that can reach the path and has the required Python packages.
5. Run metadata ingestion, then run the File Profile profile set against the folders you care about.

What You Still Need to Configure

• Data Agent selection — the source only works through an agent with local/UNC access to the path.
• Service-account permissions — read access on the path (and the share, for UNC).
• Python packages — confirm the required set is installed on the agent; add optional ones for PDF/image/encoding support.
• Folder scope — set include_sub_folders to false to keep the catalog shallow on very large trees.

Troubleshooting

Related Articles

• What Is a Data Source Template?
• AWS S3 Data Source Template
• Azure Blob Storage Data Source Template
• Python Script Data Source Template
• Macros
• Metadata Ingestion Scripts — Python Templates
• Profiling Configuration — Python Templates