---
title: "AWS S3 Data Source Template"
slug: "aws-s3-data-source-template"
updated: 2026-10-06T17:13:17Z
published: 2026-10-06T17:13:17Z
canonical: "docs.validatar.com/aws-s3-data-source-template"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.validatar.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS S3 Data Source Template

## Overview

The AWS S3 data source template turns an S3 bucket (optionally scoped to a prefix) into a Validatar data source. It lists objects with `boto3`, catalogs every object as a table, reads and profiles the common data formats (CSV, Excel, Parquet, Avro, JSON, XML, GeoJSON), and captures statistics for non-tabular objects (PDF, Markdown, images, plain text). Key prefixes become schemas, objects become tables, and columns are inferred from each object's contents.

This is a **Python (Script) template** that uses the same shared file-reading library as the [Windows Directory](/docs/windows-directory-data-source-template) and [Azure Blob Storage](/docs/azure-blob-storage-data-source-template) templates. The macros, profiling, and custom fields are identical — only the storage layer (`boto3` against S3) and the connection parameters differ.

**Platform:** Amazon S3 **Connection Category:** Script (`PythonScript`) **Runs on:** any runtime that has `boto3` and network reachability to S3 — typically the Data Agent **Template version:** 1.1.1 **Marketplace:** [AWS S3](https://admin.cloud.validatar.com/marketplace/55)

> **What's new in 1.1.1:** folder listings are scoped to the folder. Earlier versions listed by string prefix, so a folder macro or a test pointed at `logs` also returned every object under `logs-failing/` and `logs_archive/`. See [Upgrading](/docs/aws-s3-data-source-template#upgrading): the fix reaches an existing data source only after you re-select the template on it.

> **What's new in 1.1.0:** seven macros for nested JSON (see [Nested JSON](/docs/aws-s3-data-source-template#macros)); a fix for pretty-printed `.json` arrays, which were previously misdetected as line-delimited JSON and failed to read; and a fix for folder names whose first character formed a Python escape sequence.

## How It Works

- **Prefixes are schemas, objects are tables.** The template lists the bucket under the configured prefix. Each key "folder" becomes a schema (`.`-rooted, e.g. `.\raw\parquet\customer_1001`); each object becomes a table.
- **Columns are inferred on read.** Objects are read into data frames and columns/types inferred (Parquet and Avro carry explicit types; CSV and Excel are inferred).
- **One ingestion pass** catalogs the whole scope at the data-source level.
- **Scope with `bucket` + `prefix`** and control depth with `include_sub_folders`.

## Authentication

The template supports two authentication styles from a single template. Which one is used is determined by whether you fill in the access-key parameters.

| Method | How to configure | When to use |
| --- | --- | --- |
| **Access key** | Fill **Access Key ID** and **Secret Access Key** | Off-AWS runtimes, or when you want explicit, scoped credentials for Validatar |
| **IAM role / default credential chain** | Leave **Access Key ID** and **Secret Access Key** blank | When the runtime already has AWS credentials — an EC2 instance profile, ECS task role, or an agent host configured with `~/.aws/credentials` or environment variables |

With the access-key parameters blank, `boto3` resolves credentials from its standard default chain (environment variables → shared config → instance/container role). This is the recommended pattern when Validatar runs inside AWS, because there are no long-lived secrets to store or rotate.

> **Tip:** Prefer the IAM-role path in production. There are no long-lived secrets to store or rotate, and the policy is identical either way — see below.

## IAM Policy

This template is **read-only**. It never writes to S3, never deletes, and never modifies bucket configuration, so the policy it needs is small.

Attach it to whichever identity the runtime resolves: the IAM user behind the access key, or the instance profile / task role if you left the key parameters blank. The policy is the same in both cases — only the identity it hangs off differs.

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ReadObjects",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::YOUR-BUCKET/*"
    },
    {
      "Sid": "S3ListBucket",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::YOUR-BUCKET"
    }
  ]
}
```

### Why there are two different ARNs

This is the detail that causes most S3 policy problems, and it looks like a typo until you know:

- `s3:GetObject` is an **object-level** action. Its resource is `arn:aws:s3:::YOUR-BUCKET/*` — with the `/*`.
- `s3:ListBucket` and `s3:GetBucketLocation` are **bucket-level** actions. Their resource is `arn:aws:s3:::YOUR-BUCKET` — no `/*`.

Put `ListBucket` on the `/*` ARN and it silently matches nothing: ingestion completes, no error is raised, and the catalog comes back empty. Put `GetObject` on the bare bucket ARN and listing works while every read fails.

> **Note:** The IAM action is `s3:ListBucket`, but the API call it authorizes is `ListObjectsV2`. If you're reading CloudTrail to debug a denial, those are the same thing under two names.

### Scoping to a prefix

If the data source uses the `prefix` parameter, you can narrow the object permission to match:

```
{
  "Sid": "S3ReadObjects",
  "Effect": "Allow",
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::YOUR-BUCKET/data/warehouse/*"
}
```

You can also constrain listing with an `s3:prefix` condition, but do so carefully — the condition has to permit every prefix the template actually lists, including the parent when `include_sub_folders` is enabled. A prefix condition that is slightly too tight produces the same silent empty catalog as the wrong ARN. Restricting `s3:GetObject` by resource path is the simpler and more predictable control.

### Encrypted buckets

If the bucket uses **SSE-S3** (`AES256`), the policy above is sufficient — decryption is transparent.

If it uses **SSE-KMS**, reading also requires permission on the key, or every object read fails with `AccessDenied` even though the S3 permissions are correct:

```
{
  "Sid": "DecryptWithKmsKey",
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:REGION:ACCOUNT:key/YOUR-KEY-ID"
}
```

Note that `kms:GenerateDataKey` is **not** needed — that is a write-path permission, and this template only reads.

### Confirming it works

Test with the same identity Validatar will use, rather than your own:

```
aws s3 ls s3://YOUR-BUCKET/ --recursive --profile YOUR-PROFILE
```

If that lists objects, ingestion will catalog them. An `AccessDenied` here is a permissions matter, not a template one — and it is far quicker to diagnose at the CLI than through a failed ingestion run.

> **Athena is different.** If you are also using the [AWS Athena template](/docs/aws-athena-data-source-template), note that its policy is substantially larger and is **not** read-only — Athena has to write query results to S3, and it needs Glue catalog permissions plus the easily-missed `athena:GetQueryResultsStream`. Do not reuse this policy for Athena.

## Connection Parameters

| Parameter | Reference key | Type | Required | Description |
| --- | --- | --- | --- | --- |
| Bucket | `bucket` | String | Yes | S3 bucket name. |
| Prefix | `prefix` | String | No | Key prefix to scope the catalog (e.g. `data/warehouse/`). |
| Region | `region_name` | String | No | AWS region (e.g. `us-east-1`). |
| Access Key ID | `aws_access_key_id` | Secret | No | Leave blank to use an instance/agent IAM role. |
| Secret Access Key | `aws_secret_access_key` | Secret | No | Leave blank to use an instance/agent IAM role. |
| Include Sub Folders | `include_sub_folders` | Boolean | Yes | Descend into nested key prefixes. Default: `true`. |

## Required Python Packages

These must be present on the runtime that executes the source (the Data Agent, or a cloud runtime that carries the SDK).

**Required**

- `boto3` — the AWS S3 client (**required for this template**)
- `pandas`, `numpy` — the core data-frame and numeric layer
- `pyarrow` — Parquet / ORC / Feather
- `fastavro` — Avro
- `openpyxl` — Excel (`.xlsx` / `.xlsm`)

**Optional (features degrade gracefully if absent)**

- `fitz` (PyMuPDF) — PDF statistics
- `Pillow` (`PIL`) — image statistics
- `charset_normalizer` — text-encoding detection
- `lxml` — XML reading

> **Note:** The library degrades per object, not per run — a missing optional package only affects the objects that need it; the rest of the catalog continues.

## Capabilities

### Supported File Types

| Catalog file type | Extensions | Read as |
| --- | --- | --- |
| Delimited | `.csv`, `.tsv`, `.tab` | Tabular (delimiter sniffed) |
| Fixed Width / Text | `.txt`, `.flat`, `.dat` | Tabular |
| Excel | `.xlsx`, `.xlsm`, `.xls` | Tabular |
| Columnar | `.parquet`, `.orc`, `.feather` | Tabular (explicit types) |
| Row Binary | `.avro` | Tabular (explicit types) |
| Semi-Structured | `.json`, `.jsonl`, `.ndjson`, `.xml`, `.geojson` | Tabular (flattened / normalized) |
| Document | `.pdf`, `.md`, `.markdown`, `.rst` | Content statistics |
| Image | `.png`, `.jpg`, `.jpeg`, `.gif`, `.bmp`, `.tif`, `.tiff`, `.webp` | Content statistics |

> **Note:** Line-delimited JSON is detected even when the object has a `.json` extension — common for S3 export jobs — and flattened into columns.

### Macros

The template ships with 28 macros: 21 generic file macros and seven for nested JSON. Because every macro goes through the shared reader, the same test works against any object and any supported format. Parameters are metadata-linked (Folder → Schema, File → Table, Column) so they render as catalog-driven dropdowns.

**A. Single-object data**

| Macro | Returns |
| --- | --- |
| File - Read Data (any type) | The whole object as a dataset |
| File - First N Rows | The first N rows (default 100) |
| File - Row Count | Row count as a single value |
| File - Schema (columns + inferred types) | Column list with inferred type and null % |
| File - Profile One Column | Profile statistics for one column |
| File - Row Count Grouped by Column | Group-by counts |
| File - Aggregate a Column | `sum` / `min` / `max` / `mean` / `count` / `nunique` |

**B. Pattern & union (multi-object)**

| Macro | Returns |
| --- | --- |
| Folder - Files Matching a Pattern | Objects whose name matches a glob (e.g. `account_*.parquet`) |
| Folder - Union Files Matching a Pattern | Every matching object read and unioned (schema-aligned, source-tagged) |
| Folder - Row Count Across Matching Files | Per-object and total row counts |
| Folder - Schema Consistency Across Files | Each object's columns and whether they match the first (partitioned extracts) |

**C. Prefix & object information**

| Macro | Returns |
| --- | --- |
| Folder - File Inventory | Every object with size, detected type, and modified date |
| Folder - Summary | One-row rollup: object count, total size, distinct types, date range |
| File - Full Metadata | One object's size, modified date, MD5, and (if tabular) row/column counts |

**D. Content statistics (including non-tabular)**

| Macro | Returns |
| --- | --- |
| File - Content Stats (any type) | Type-aware stats for any object |
| PDF - Statistics | Pages, words, embedded images, title/author, encryption flag |
| Markdown - Statistics | Headings, links, images, code blocks, table rows |
| Text - Statistics | Lines, words, encoding, blank lines, longest line |
| Image - Statistics | Width/height, megapixels, format, mode, alpha |
| JSON - Statistics | Record count, nesting depth, top-level keys |
| XML - Statistics | Root tag, element count, distinct tags, depth |

> **Tip:** **Folder - Union Files Matching a Pattern** is ideal for partitioned S3 layouts — point it at a prefix of `*.parquet` files and it returns one unified dataset with a `_source_file` column, so you can validate the combined set and still trace any row to its object.

**E. Nested JSON**

The seven macros above deal with JSON as a *flat* table. These seven deal with it as a *tree*.

The distinction matters more than it sounds. The standard reader normalizes nested **objects** into dotted columns — `customer.address.city` works fine. But a nested **array** cannot become a column, so it stays in the cell as a raw list:

```
order_id     customer.address.city   items
ORD-00001    Austin                  [{'sku': 'WID-001', 'quantity': 3, ...}, {...}]
```

You cannot count that, group it, join on it, or assert anything about it. For order payloads, event streams, API captures — anything where the interesting data is inside a repeated element — that is where testing usually stops.

| Macro | Returns |
| --- | --- |
| JSON - Discover Nested Structure | One row per path in the document tree: depth, JSON type(s), how many records contain it, array cardinality, and a sample value |
| JSON - Flatten to Table | The whole document as a table, with configurable array handling (`join` / `explode` / `count`) |
| JSON - Explode an Array Path | One row per element of a chosen array, optionally carrying the parent record's fields |
| JSON - Extract Specific Paths | A chosen set of dotted paths as columns; `[]` descends into every array element |
| JSON - Array Cardinality per Record | Element count per record for a chosen array path, keyed by a business identifier |
| JSON - Type Conflicts Across Records | Paths whose JSON type varies between records |
| JSON - Key Coverage / Sparse Fields | Paths present in fewer than N% of records |

**Start with Discover Nested Structure.** It is the map — run it once against a sample object and it tells you every path, how deep it is, and which ones are arrays, so you know what to feed the other six.

**Then reach for Explode an Array Path, not Flatten to Table.** Both produce tabular output, but flattening with `arrays=explode` expands *every* array at once, and sibling arrays multiply: a payload with 3 line items and 2 contacts yields 6 rows, and any sum over it is wrong by a factor of 2. Exploding one path does not have that problem. Flatten's default is therefore `join`, which keeps one row per record and encodes arrays as text.

> **Note:** `explode` is the right choice when you genuinely want the cross-product, or when only one array is present. The parameter exists so the behaviour is a decision rather than a surprise.

See [Macros](/docs/data-source-template-macros) for how macros are used when building tests, and the **AWS S3 — Nested JSON Sample Tests** pack for six worked examples, including one that joins a SKU two levels deep inside an array to a column in an Excel file.

### Metadata Ingestion

Ingestion runs once at the data-source level. It lists the bucket under the prefix, registers each key prefix as a schema and each object as a table, and populates the **File Info** catalog custom fields — including per-prefix rollups (object count, total size, distinct file types, newest/oldest modified). Column-level metadata is discovered on demand by the profile set.

See [Metadata Ingestion Scripts — Python Templates](/docs/metadata-ingestion-scripts-python-templates).

### Profiling

The template ships one profile: **File Profile (columns + table metrics, any file type)** — the same type-safe profiler as the Windows Directory template. Run it as a profile set against a prefix to profile every object in it. Following the principle that *numeric/statistical values are profiles and descriptors are custom fields*, it produces:

- **Column-level profiles:** null/blank/distinct counts and percentages, length statistics, most-common value, numeric statistics (min/max/mean/median/standard deviation, zero/negative counts), and date range.
- **Table-level file metrics:** file size (bytes), file age (days), row count, column count, and type-specific metrics (PDF page count, content word/line/record count, image width/height).

See [Profiling Configuration — Python Templates](/docs/profiling-configuration-python-templates).

### Catalog Custom Fields (required dependency)

The ingestion script emits **File Info** catalog custom fields to describe objects and prefixes — the text/tag/date descriptors and prefix rollups that don't fit the numeric profile model. Import the **File Info - Catalog Custom Fields** set **before** running ingestion; Validatar silently drops custom-field columns with no matching definition.

**Schema level (prefix rollups):** Contains Files, Schema Object Represents, Full Path, File Count, Total Size (bytes), Distinct File Types, Readable File Count, Newest Modified, Oldest Modified

**Table level (per object):** File Type, File Name, Sheet Name, File Size, File Total Bytes, File Extension, Date Modified, File Logical Table

> **Warning:** Catalog custom fields import through the UI only (**Settings > Catalog Custom Fields > Import**) and require the **GlobalCustomMetadataAdmin** role.

## Installation

1. **Import the custom fields first** — the **File Info - Catalog Custom Fields** set (**Settings > Catalog Custom Fields > Import**).
2. **Import the template** — **Settings > Data Source Templates > Import**, select the AWS S3 template file.
3. **Create a data source** and set **Bucket**, **Prefix**, and **Region**. For access-key auth, add the two secret keys; for IAM-role auth, leave them blank.
4. **Assign a runtime** with `boto3` and network access to S3.
5. **Run metadata ingestion**, then run the **File Profile** profile set.

## Upgrading

Importing a newer version of the template with **Overwrite existing** updates the template itself and nothing else. Each data source built from a Script template keeps its own copy of the template's pre-execution script, taken at the moment the template was selected. A data source created on 1.1.0 keeps running the 1.1.0 code until you refresh that copy.

To pick up a new version on an existing data source:

1. Import the new template file (**Settings > Data Source Templates > Import**, with overwrite).
2. Open each data source built from the template and edit its connection.
3. Re-select **AWS S3** as the template, confirm the parameters are still filled, and save.
4. Run a folder macro (for example **Folder - File Inventory** on a prefix that has a similarly named neighbour) to confirm the new behaviour.

New data sources created after the import pick up the current version with no extra step.

## What You Still Need to Configure

- **Credentials strategy** — access keys vs. IAM role. Prefer the role when Validatar runs in AWS.
- **IAM permissions** — read on the target bucket/prefix (`s3:GetObject`, `s3:ListBucket`).
- **Runtime placement** — a Data Agent or cloud runtime that carries `boto3`.
- **Scope** — set `prefix` to avoid cataloging an entire large bucket, and `include_sub_folders` to control depth.

## Troubleshooting

| Symptom | Cause | Fix |
| --- | --- | --- |
| `AccessDenied` on list/read | Resolved identity lacks bucket permission | See [IAM Policy](/docs/aws-s3-data-source-template#iam-policy) — check the object vs bucket ARNs first |
| Ingestion completes but the catalog is empty | `s3:ListBucket` granted on the object ARN (`/*`) instead of the bucket ARN | Bucket-level actions take the bare bucket ARN — see [IAM Policy](/docs/aws-s3-data-source-template#iam-policy) |
| Listing works, every object read fails | `s3:GetObject` granted on the bucket ARN instead of the object ARN | Object-level actions need the `/*` suffix |
| `AccessDenied` on read only, S3 permissions look correct | Bucket is SSE-KMS encrypted and the identity has no key permission | Add `kms:Decrypt` + `kms:DescribeKey` on the CMK |
| "partial credentials found, missing aws_secret_access_key" | Only one of the two access-key parameters is filled | Provide both, or leave both blank for IAM-role auth |
| Empty catalog | Wrong bucket/prefix, or region mismatch | Verify the bucket, prefix, and region |
| A single object errors | Missing optional package for that format | Install the package (`fastavro`, `openpyxl`, `pyarrow`, `fitz`, `Pillow`) |
| Custom-field values missing | File Info custom fields not imported before ingestion | Import the File Info set, then re-run ingestion |
| `Columnar read failed … Repetition level histogram size mismatch` | The Parquet file was written by a **newer** `pyarrow` than the runtime has | Upgrade `pyarrow` on the Data Agent — see below |
| A folder's objects report `NoSuchKey` even though they exist | Folder name begins with a character that forms a Python escape (`reference`, `temp`, `feeds`, `nightly`, …) | Fixed in template 1.1.0; upgrade the template |
| A folder macro or test returns objects from a neighbouring prefix (`logs` also lists `logs-failing/…`) | Versions before 1.1.1 listed by string prefix rather than by folder | Fixed in template 1.1.1; import it and re-select the template on the data source (see [Upgrading](/docs/aws-s3-data-source-template#upgrading)) |
| The template was upgraded but the old behaviour persists | The data source still holds its own copy of the previous pre-execution script | Re-select the template on the data source (see [Upgrading](/docs/aws-s3-data-source-template#upgrading)) |

### Parquet writer/reader version skew

Parquet is not as version-portable as it looks. A file written by `pyarrow` 20 or newer records **size statistics** that older readers reject outright:

```
Columnar read failed for sales_fact.parquet: Repetition level histogram size mismatch
```

This is a *reader* limitation, and no writer setting avoids it — format version (`1.0`, `2.4`, `2.6`), `data_page_version`, `store_schema`, `write_statistics`, `use_dictionary` and `flavor="spark"` all still produce a file the older reader refuses. `pyarrow` exposes no size-statistics toggle in its Python API.

If you hit this, **upgrade `pyarrow` on the Data Agent host** to at least the version that wrote your files. It is worth checking proactively: modern Spark, pandas and dbt all write Parquet with recent Arrow, so a lagging agent runtime can fail on files that every other tool in your stack reads without complaint.

To see what your agent actually has, run a one-line Python test against the data source:

```
import pyarrow, pandas, sys
dataframe = pd.DataFrame({"VALUE": [f"py={sys.version.split()[0]} pyarrow={pyarrow.__version__} pandas={pandas.__version__}"]})
```

## Sample Test Packs

Two marketplace packs ship worked examples against this template:

- **AWS S3 - Unstructured File Sample Tests** ([marketplace](https://admin.cloud.validatar.com/marketplace/64)): 190 tests on text, log and markdown files, PDFs and images, each with its adjustable values in a block at the top of the script. See [AWS S3 Unstructured File Sample Tests](/docs/aws-s3-unstructured-sample-tests). The files those tests read are a separate download, [Unstructured File Sample Fixture (S3 and Azure Blob)](https://admin.cloud.validatar.com/marketplace/66).
- **AWS S3 - Nested JSON Sample Tests**: six tests that reach through arrays of objects in an order payload, including a join from a SKU inside an array to a column in an Excel file.

## Related Articles

- [What Is a Data Source Template?](/docs/what-is-a-data-source-template-overview)
- [AWS S3 Unstructured File Sample Tests](/docs/aws-s3-unstructured-sample-tests)
- [Windows Directory Data Source Template](/docs/windows-directory-data-source-template)
- [Azure Blob Storage Data Source Template](/docs/azure-blob-storage-data-source-template)
- [Python Script Data Source Template](/docs/python-script-data-source-template)
- [Macros](/docs/data-source-template-macros)
- [Metadata Ingestion Scripts — Python Templates](/docs/metadata-ingestion-scripts-python-templates)
- [Profiling Configuration — Python Templates](/docs/profiling-configuration-python-templates)