AWS S3 Data Source Template

Overview

The AWS S3 data source template turns an S3 bucket (optionally scoped to a prefix) into a Validatar data source. It lists objects with boto3, catalogs every object as a table, reads and profiles the common data formats (CSV, Excel, Parquet, Avro, JSON, XML, GeoJSON), and captures statistics for non-tabular objects (PDF, Markdown, images, plain text). Key prefixes become schemas, objects become tables, and columns are inferred from each object's contents.

This is a Python (Script) template that uses the same shared file-reading library as the Windows Directory and Azure Blob Storage templates. The macros, profiling, and custom fields are identical — only the storage layer (boto3 against S3) and the connection parameters differ.

Platform: Amazon S3
Connection Category: Script (PythonScript)
Runs on: any runtime that has boto3 and network reachability to S3 — typically the Data Agent
Template version: 1.0.0

How It Works

• Prefixes are schemas, objects are tables. The template lists the bucket under the configured prefix. Each key "folder" becomes a schema (.-rooted, e.g. .\raw\parquet\customer_1001); each object becomes a table.
• Columns are inferred on read. Objects are read into data frames and columns/types inferred (Parquet and Avro carry explicit types; CSV and Excel are inferred).
• One ingestion pass catalogs the whole scope at the data-source level.
• Scope with bucket + prefix and control depth with include_sub_folders.

Authentication

The template supports two authentication styles from a single template. Which one is used is determined by whether you fill in the access-key parameters.

With the access-key parameters blank, boto3 resolves credentials from its standard default chain (environment variables → shared config → instance/container role). This is the recommended pattern when Validatar runs inside AWS, because there are no long-lived secrets to store or rotate.

Tip: Prefer the IAM-role path in production. Grant the runtime's role read-only access to just the bucket/prefix Validatar needs (s3:GetObject, s3:ListBucket scoped to that ARN). An AccessDenied while listing or reading almost always means the resolved identity lacks permission on the bucket — a permissions matter, not a template one.

Connection Parameters

Required Python Packages

These must be present on the runtime that executes the source (the Data Agent, or a cloud runtime that carries the SDK).

Required

• boto3 — the AWS S3 client (required for this template)
• pandas, numpy — the core data-frame and numeric layer
• pyarrow — Parquet / ORC / Feather
• fastavro — Avro
• openpyxl — Excel (.xlsx / .xlsm)

Optional (features degrade gracefully if absent)

• fitz (PyMuPDF) — PDF statistics
• Pillow (PIL) — image statistics
• charset_normalizer — text-encoding detection
• lxml — XML reading

Note: The library degrades per object, not per run — a missing optional package only affects the objects that need it; the rest of the catalog continues.

Capabilities

Supported File Types

Note: Line-delimited JSON is detected even when the object has a .json extension — common for S3 export jobs — and flattened into columns.

Macros

The template ships with 21 generic macros. Because every macro goes through the shared reader, the same test works against any object and any supported format. Parameters are metadata-linked (Folder → Schema, File → Table, Column) so they render as catalog-driven dropdowns.

A. Single-object data

B. Pattern & union (multi-object)

C. Prefix & object information

D. Content statistics (including non-tabular)

Tip: Folder - Union Files Matching a Pattern is ideal for partitioned S3 layouts — point it at a prefix of *.parquet files and it returns one unified dataset with a _source_file column, so you can validate the combined set and still trace any row to its object.

See Macros for how macros are used when building tests.

Metadata Ingestion

Ingestion runs once at the data-source level. It lists the bucket under the prefix, registers each key prefix as a schema and each object as a table, and populates the File Info catalog custom fields — including per-prefix rollups (object count, total size, distinct file types, newest/oldest modified). Column-level metadata is discovered on demand by the profile set.

See Metadata Ingestion Scripts — Python Templates.

Profiling

The template ships one profile: File Profile (columns + table metrics, any file type) — the same type-safe profiler as the Windows Directory template. Run it as a profile set against a prefix to profile every object in it. Following the principle that numeric/statistical values are profiles and descriptors are custom fields, it produces:

• Column-level profiles: null/blank/distinct counts and percentages, length statistics, most-common value, numeric statistics (min/max/mean/median/standard deviation, zero/negative counts), and date range.
• Table-level file metrics: file size (bytes), file age (days), row count, column count, and type-specific metrics (PDF page count, content word/line/record count, image width/height).

See Profiling Configuration — Python Templates.

Catalog Custom Fields (required dependency)

The ingestion script emits File Info catalog custom fields to describe objects and prefixes — the text/tag/date descriptors and prefix rollups that don't fit the numeric profile model. Import the File Info - Catalog Custom Fields set before running ingestion; Validatar silently drops custom-field columns with no matching definition.

Schema level (prefix rollups): Contains Files, Schema Object Represents, Full Path, File Count, Total Size (bytes), Distinct File Types, Readable File Count, Newest Modified, Oldest Modified

Table level (per object): File Type, File Name, Sheet Name, File Size, File Total Bytes, File Extension, Date Modified, File Logical Table

Warning: Catalog custom fields import through the UI only (Settings > Catalog Custom Fields > Import) and require the GlobalCustomMetadataAdmin role.

Installation

1. Import the custom fields first — the File Info - Catalog Custom Fields set (Settings > Catalog Custom Fields > Import).
2. Import the template — Settings > Data Source Templates > Import, select the AWS S3 template file.
3. Create a data source and set Bucket, Prefix, and Region. For access-key auth, add the two secret keys; for IAM-role auth, leave them blank.
4. Assign a runtime with boto3 and network access to S3.
5. Run metadata ingestion, then run the File Profile profile set.

What You Still Need to Configure

• Credentials strategy — access keys vs. IAM role. Prefer the role when Validatar runs in AWS.
• IAM permissions — read on the target bucket/prefix (s3:GetObject, s3:ListBucket).
• Runtime placement — a Data Agent or cloud runtime that carries boto3.
• Scope — set prefix to avoid cataloging an entire large bucket, and include_sub_folders to control depth.

Troubleshooting

Related Articles

• What Is a Data Source Template?
• Windows Directory Data Source Template
• Azure Blob Storage Data Source Template
• Python Script Data Source Template
• Macros
• Metadata Ingestion Scripts — Python Templates
• Profiling Configuration — Python Templates