---
title: "Windows Directory Data Source Template"
slug: "windows-directory-data-source-template"
updated: 2026-09-29T20:28:11Z
published: 2026-09-29T20:28:11Z
canonical: "docs.validatar.com/windows-directory-data-source-template"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.validatar.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Windows Directory Data Source Template

## Overview

The Windows Directory data source template turns a local or network directory tree into a Validatar data source. It walks the folder tree, catalogs every file as a table, reads and profiles the common data formats (CSV, Excel, Parquet, Avro, JSON, XML, GeoJSON), and captures statistics for non-tabular files (PDF, Markdown, images, plain text). Folders become schemas, files become tables, and columns are inferred from each file's contents.

This is a **Python (Script) template**. All discovery and reading is done by Python running on the Data Agent, so the same template works whether the data lives on the agent's local disk or on a UNC share the agent can reach.

**Platform:** Windows file system (local paths and UNC shares) **Connection Category:** Script (`PythonScript`) **Runs on:** the Data Agent (required — it reads the local/UNC file system) **Template version:** 2.1.0 **Marketplace:** [Windows Directory](https://admin.cloud.validatar.com/marketplace/45)

> **Note:** The Windows Directory, [AWS S3](/docs/aws-s3-data-source-template), and [Azure Blob Storage](/docs/azure-blob-storage-data-source-template) templates are built from one shared file-reading library. They expose the same macros, profiling, and custom fields — only the storage layer (local disk vs. S3 vs. Blob) and the connection parameters differ. Learn one and you know all three.

## How It Works

- **Folders are schemas, files are tables.** The template walks from the root folder you configure. Each folder (relative to the root, `.`-rooted — `.`, `.\CSV`, `.\CSV\CUSTOMER_1001`) becomes a schema; each file inside it becomes a table.
- **Columns are inferred on read.** For tabular formats, the template reads the file and infers column names and data types from the content (Parquet and Avro carry explicit types; CSV and Excel are inferred).
- **One ingestion pass.** Metadata ingestion runs at the data-source level and catalogs the whole tree in a single pass.
- **Everything is driven by two parameters** — the root path and whether to descend into subfolders.

## Authentication

The Windows Directory template has **no credential parameters**. Access is governed entirely by the **Windows identity the Data Agent runs as** (its service account). There is nothing to enter in Validatar beyond the path.

| Access scenario | Requirement |
| --- | --- |
| Local disk on the agent host | The agent's service account needs read access to the folder. |
| UNC / network share (`\\server\share\...`) | The agent's service account must be a domain (or trusted) account with read permission on the share **and** the underlying NTFS folder. |

> **Tip:** If ingestion returns an empty catalog or "access denied," it is almost always the agent's service account lacking read permission on the path — not a Validatar configuration issue. Confirm the account can open the path in Windows Explorer while logged in as that identity.

## Connection Parameters

| Parameter | Reference key | Type | Required | Description |
| --- | --- | --- | --- | --- |
| Parent Folder Path | `parent_folder_path` | String | Yes | Root directory to catalog. Local path (`D:\data\exports`) or UNC share (`\\fileserver\finance\extracts`). |
| Include Sub Folders | `include_sub_folders` | Boolean | Yes | When true, descends into every subfolder (each becomes a schema). When false, only the root folder is cataloged. Default: `true`. |

## Required Python Packages

These must be present on the Data Agent's Python runtime.

**Required**

- `pandas`, `numpy` — the core data-frame and numeric layer (hard requirement)
- `pyarrow` — Parquet / ORC / Feather
- `fastavro` — Avro
- `openpyxl` — Excel (`.xlsx` / `.xlsm`)

**Optional (features degrade gracefully if absent)**

- `fitz` (PyMuPDF) — PDF statistics
- `Pillow` (`PIL`) — image statistics
- `charset_normalizer` — best-effort text-encoding detection
- `lxml` — XML reading

> **Note:** The library degrades per file, not per run. If an optional package is missing, only the files that need it are affected — that file returns a clear error and the rest of the catalog continues. Missing `fastavro`, for example, doesn't stop your CSVs from being read and profiled.

## Capabilities

### Supported File Types

| Catalog file type | Extensions | Read as |
| --- | --- | --- |
| Delimited | `.csv`, `.tsv`, `.tab` | Tabular (delimiter sniffed) |
| Fixed Width / Text | `.txt`, `.flat`, `.dat` | Tabular (fixed-width inference, else one line per row) |
| Excel | `.xlsx`, `.xlsm`, `.xls` | Tabular |
| Columnar | `.parquet`, `.orc`, `.feather` | Tabular (explicit types) |
| Row Binary | `.avro` | Tabular (explicit types) |
| Semi-Structured | `.json`, `.jsonl`, `.ndjson`, `.xml`, `.geojson` | Tabular (flattened / normalized) |
| Document | `.pdf`, `.md`, `.markdown`, `.rst` | Content statistics |
| Image | `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.tif`, `.tiff`, `.webp` | Content statistics |

> **Note:** JSON is handled flexibly — arrays of objects, a single object, an object of arrays, GeoJSON feature collections, and line-delimited JSON (even when saved with a `.json` extension) are all detected and flattened into columns.

### Macros

The template ships with 28 macros: 21 generic file macros and seven for nested JSON. Every generic macro works on every file type through the shared reader, so you write a test once and point it at any folder and file. Macros take metadata-linked parameters (Folder → Schema, File → Table, Column) so they render as catalog-driven dropdowns when you build a test.

**A. Single-file data**

| Macro | Returns |
| --- | --- |
| File - Read Data (any type) | The whole file as a dataset |
| File - First N Rows | The first N rows (default 100) |
| File - Row Count | Row count as a single value |
| File - Schema (columns + inferred types) | Column list with inferred type and null % |
| File - Profile One Column | Profile statistics for one column |
| File - Row Count Grouped by Column | Group-by counts |
| File - Aggregate a Column | `sum` / `min` / `max` / `mean` / `count` / `nunique` |

**B. Pattern & union (multi-file)**

| Macro | Returns |
| --- | --- |
| Folder - Files Matching a Pattern | Files whose name matches a glob (e.g. `account_*.csv`) |
| Folder - Union Files Matching a Pattern | Every matching file read and unioned (schema-aligned, each row tagged with its source file) |
| Folder - Row Count Across Matching Files | Per-file and total row counts |
| Folder - Schema Consistency Across Files | Each file's columns and whether they match the first file (daily/partitioned extracts) |

**C. Folder & file information**

| Macro | Returns |
| --- | --- |
| Folder - File Inventory | Every file with size, detected type, and modified date |
| Folder - Summary | One-row rollup: file count, total size, distinct types, date range |
| File - Full Metadata | One file's size, modified date, MD5, and (if tabular) row/column counts |

**D. Content statistics (including non-tabular)**

| Macro | Returns |
| --- | --- |
| File - Content Stats (any type) | Type-aware stats for any file |
| PDF - Statistics | Page count, word count, embedded images, title/author, encryption flag |
| Markdown - Statistics | Headings, links, images, code blocks, table rows |
| Text - Statistics | Lines, words, encoding, blank lines, longest line |
| Image - Statistics | Width/height, megapixels, format, mode, alpha |
| JSON - Statistics | Record count, nesting depth, top-level keys |
| XML - Statistics | Root tag, element count, distinct tags, depth |

> **Tip:** **Folder - Union Files Matching a Pattern** is the workhorse for partitioned or daily extracts — point it at a folder of `*.parquet` or `*.avro` files and it returns one unified dataset with a `_source_file` column, so you can test the combined set and still trace any row back to its file.

**E. Nested JSON**

The macros above treat JSON as a flat table. These seven treat it as a tree.

The difference matters more than it sounds. The standard reader normalizes nested objects into dotted columns, so `customer.address.city` works as a column. A nested array cannot become a column, so it stays in the cell as a raw list:

```
order_id     customer.address.city   items
ORD-00001    Austin                  [{'sku': 'WID-001', 'quantity': 3, ...}, {...}]
```

You cannot count that, group it, join on it, or assert anything about it. For order payloads, event streams and API captures, where the interesting data is inside a repeated element, that is where testing usually stops.

| Macro | Returns |
| --- | --- |
| JSON - Discover Nested Structure | One row per path in the document tree: depth, JSON type(s), how many records contain it, array cardinality, and a sample value |
| JSON - Flatten to Table | The whole document as a table, with configurable array handling (`join` / `explode` / `count`) |
| JSON - Explode an Array Path | One row per element of a chosen array, optionally carrying the parent record's fields |
| JSON - Extract Specific Paths | A chosen set of dotted paths as columns; `[]` descends into every array element |
| JSON - Array Cardinality per Record | Element count per record for a chosen array path, keyed by a business identifier |
| JSON - Type Conflicts Across Records | Paths whose JSON type varies between records |
| JSON - Key Coverage / Sparse Fields | Paths present in fewer than N% of records |

**Start with Discover Nested Structure.** It is the map. Run it once against a sample file in the folder and it tells you every path, how deep it is, and which ones are arrays, so you know what to feed the other six.

**Then reach for Explode an Array Path, not Flatten to Table.** Both produce tabular output, but flattening with `arrays=explode` expands every array at once, and sibling arrays multiply: a record with 3 line items and 2 contacts yields 6 rows, and any sum over it is wrong by a factor of 2. Exploding one path does not have that problem. Flatten's default is therefore `join`, which keeps one row per record and encodes arrays as text.

> **Note:** `explode` is the right choice when you want the cross-product, or when only one array is present. The parameter exists so the behaviour is a decision rather than a surprise.

The same 28 macros ship in the [AWS S3](/docs/aws-s3-data-source-template) and [Azure Blob Storage](/docs/azure-blob-storage-data-source-template) templates, so a test written against a local folder runs unchanged against a bucket or a container.

See [Macros](/docs/data-source-template-macros) for how macros are used when building tests.

### Metadata Ingestion

Ingestion runs once at the data-source level. It walks the tree, registers each folder as a schema and each file as a table, and populates the **File Info** catalog custom fields (see below) — including per-folder rollups (file count, total size, distinct file types, newest/oldest modified). Column-level metadata is discovered on demand by the profile set rather than during ingestion, which keeps ingestion fast on large trees.

See [Metadata Ingestion Scripts — Python Templates](/docs/metadata-ingestion-scripts-python-templates) for how Python ingestion scripts work.

### Profiling

The template ships one profile: **File Profile (columns + table metrics, any file type)**. It runs per table (file) and is type-safe — it reads any supported file and emits only the metrics that apply to that file's type. Run it as a profile set against a folder to profile every file in it.

Following the principle that *anything numeric or statistical about an asset is a profile, and descriptors/tags/text belong in custom fields*, it produces:

**Column-level profiles** (numeric, per column)

- Null count / percent, blank count / percent, distinct count / percent
- Length statistics: min / max / mean length
- Most common value and its count
- Numeric statistics (numeric columns): min, max, mean, median, standard deviation, zero count, negative count
- Date range (date columns): min date, max date

**Table-level file metrics** (numeric, per file)

- File size (bytes), file age (days), row count, column count
- Type-specific: PDF page count, content word/line/record count, image width/height (px)

> **Note:** The column profiler is deliberately type-safe: a mixed-type or all-text column never breaks the run — non-applicable metrics are simply omitted for that column.

See [Profiling Configuration — Python Templates](/docs/profiling-configuration-python-templates) for profile-set mechanics.

### Catalog Custom Fields (required dependency)

The ingestion script emits catalog custom fields to describe files and folders. These fields hold **descriptors** — text, tags, dates, and folder rollups — that don't fit the numeric profile model (schemas in particular can't be profiled, so folder rollups are captured here).

You must import the **File Info - Catalog Custom Fields** set **before** you run ingestion. Validatar silently drops custom-field columns that have no matching field definition, so without it these values are simply lost.

**Schema level (folder rollups):** Contains Files, Schema Object Represents, Full Path, File Count, Total Size (bytes), Distinct File Types, Readable File Count, Newest Modified, Oldest Modified

**Table level (per file):** File Type, File Name, Sheet Name, File Size, File Total Bytes, File Extension, Date Modified, File Logical Table

> **Warning:** Catalog custom fields import through the UI only (**Settings > Catalog Custom Fields > Import**), and require the **GlobalCustomMetadataAdmin** role. Import the File Info set first, then import this template, then create the data source and run ingestion.

## Installation

1. **Import the custom fields first.** In **Settings > Catalog Custom Fields > Import**, import the **File Info - Catalog Custom Fields** set.
2. **Import the template.** In **Settings > Data Source Templates > Import**, select the Windows Directory template file.
3. **Create a data source** from the template and set **Parent Folder Path** and **Include Sub Folders**.
4. **Assign a Data Agent** that can reach the path and has the required Python packages.
5. **Run metadata ingestion**, then run the **File Profile** profile set against the folders you care about.

## Upgrading

Importing a newer version of the template with **Overwrite existing** updates the template itself and nothing else. Each data source built from a Script template keeps its own copy of the template's pre-execution script, taken at the moment the template was selected. A data source created on 2.0.0 keeps running the 2.0.0 code until you refresh that copy.

To pick up a new version on an existing data source:

1. Import the new template file (**Settings > Data Source Templates > Import**, with overwrite).
2. Open each data source built from the template and edit its connection.
3. Re-select **Windows Directory** as the template, confirm the parameters are still filled, and save.
4. Run **JSON - Discover Nested Structure** against any JSON file in the tree. On a data source still carrying the old pre-execution script it fails with a `NameError` for the JSON helpers; on a refreshed one it returns the path listing.

New data sources created after the import pick up the current version with no extra step.

## What You Still Need to Configure

- **Data Agent selection** — the source only works through an agent with local/UNC access to the path.
- **Service-account permissions** — read access on the path (and the share, for UNC).
- **Python packages** — confirm the required set is installed on the agent; add optional ones for PDF/image/encoding support.
- **Folder scope** — set `include_sub_folders` to false to keep the catalog shallow on very large trees.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| Empty catalog after ingestion | Agent service account can't read the path | Grant read permission to the agent's identity on the folder / UNC share |
| A single file errors, others succeed | Missing optional package for that format | Install the package (`fastavro`, `openpyxl`, `pyarrow`, `fitz`, `Pillow`) |
| Custom-field values missing in the catalog | File Info custom fields not imported before ingestion | Import the File Info set, then re-run ingestion |
| "not a tabular file type" from a data macro | Macro used on a PDF/image/text file | Use a content-stats macro (PDF/Image/Text/JSON/XML Statistics) instead |
| UNC path works locally but not from the agent | Agent running as `LocalSystem` or a local account | Run the agent service as a domain account with share access |

## Sample Test Pack

**Windows Directory - Unstructured File Sample Tests** ([marketplace](https://admin.cloud.validatar.com/marketplace/67)) ships 190 worked examples against this template: text, log and markdown files, PDFs and images, each with its adjustable values in a block at the top of the script and a switch that points it at a file built to make it fail. See [Windows Directory Unstructured File Sample Tests](/docs/windows-directory-unstructured-sample-tests). The files those tests read are a separate download, [Unstructured File Sample Fixture (S3 and Azure Blob)](https://admin.cloud.validatar.com/marketplace/66); unzip it and copy the six top-level folders under the data source's Parent Folder Path on the Data Agent host.

## Related Articles

- [What Is a Data Source Template?](/docs/what-is-a-data-source-template-overview)
- [Windows Directory Unstructured File Sample Tests](/docs/windows-directory-unstructured-sample-tests)
- [AWS S3 Data Source Template](/docs/aws-s3-data-source-template)
- [Azure Blob Storage Data Source Template](/docs/azure-blob-storage-data-source-template)
- [Python Script Data Source Template](/docs/python-script-data-source-template)
- [Macros](/docs/data-source-template-macros)
- [Metadata Ingestion Scripts — Python Templates](/docs/metadata-ingestion-scripts-python-templates)
- [Profiling Configuration — Python Templates](/docs/profiling-configuration-python-templates)