Replication of Your Metadata to a Data Processing Engine

Overview

Validatar stores all of its operational data — metadata catalogs, test definitions, execution results, profiling metrics, trust scores, and more — in its internal repository database (SQL Server or PostgreSQL). While this database powers the application, it isn't designed for ad-hoc analytics or cross-platform reporting.

Repository replication solves this by continuously syncing Validatar's internal data to a Snowflake data processing engine. Once configured, Validatar incrementally loads repository tables into a dedicated Snowflake schema, giving your team direct SQL access to all of Validatar's operational data. This enables custom reporting, integration with BI tools, cross-environment comparisons, and any analysis that benefits from having Validatar's data in a queryable warehouse.

Prerequisites

Before configuring repository replication, you need:

• A Snowflake data processing engine already created in Validatar (see Creating Data Processing Engines)
• The Snowflake Results Storage license feature enabled for your Validatar environment
• Administrator access — only users with the Global Configuration Admin role can configure replication

Configuring Repository Replication

Navigate to Settings > Configuration > Repository Replication.

Step 1: Select a Snowflake Data Processing Engine

Choose the Snowflake data processing engine that will receive the replicated data. The dropdown lists all verified Snowflake engines configured in your environment.

Step 2: Specify a Schema

Enter the Snowflake schema name where replicated tables will be created. The default is REPOSITORY.

Note: The replication schema must be different from the default schema of the selected data processing engine. Validatar creates and manages this schema automatically — you don't need to create it in Snowflake ahead of time.

Step 3: Set a Sync Schedule

Click Set Sync Schedule to configure how often replication runs.

You can schedule replication to run:

• Daily — once per day or multiple times per day
• Weekly — on specific days of the week
• Monthly — on specific days of the month

For most organizations, a daily sync is sufficient. If your team needs near-real-time access to test results or execution history, consider scheduling multiple runs per day.

Step 4: Save

Click Save to persist the configuration. Replication will begin according to the schedule you set.

How Incremental Loading Works

Repository replication uses an incremental loading strategy to minimize the data transferred on each run.

The Watermark Approach

Each replicated table tracks a watermark — the timestamp of the last successful sync, stored as DateLastSynced per table. On each replication run:

1. Validatar queries its internal database for rows where the sync column is greater than the watermark
2. Only new and modified rows since the last sync are selected
3. These rows are serialized, streamed to a Snowflake staging table, and merged into the target tables
4. The watermark advances to the latest timestamp seen

The sync column is DateModified when available, falling back to DateCreated for tables that don't track modification timestamps.

The Staging and Merge Process

Replication uses a staging-then-merge pattern:

1. Staging — Changed rows are serialized as JSON and uploaded to a transient staging table (RepositoryReplicationStaging) in Snowflake
2. Merge — For each table, Validatar executes a MERGE INTO statement that:
   • Updates existing rows (matched on the primary key ID)
   • Inserts new rows that don't exist in the target
3. Cleanup — Staging records are deleted after a successful merge

This approach is efficient (only changed data is transferred), idempotent (re-running the same batch produces the same result), and handles both inserts and updates in a single operation.

Full Re-Sync

If you need to force a complete reload of specific tables — for example, after a schema change or data correction — you can reset the sync dates for individual tables. This clears the watermark and causes the next replication run to pull all rows for those tables.

What Gets Replicated

Validatar replicates 55 tables from its internal repository, covering every major area of the platform. All table names in Snowflake use SNAKE_CASE naming convention.

Metadata Catalog

Profiling and Trust Scores

Tests and Executions

Template Tests

Jobs and Scheduling

Projects and Permissions

Connections

Custom Fields

Users and Roles

Configuration and System

Helper Views

After each replication run, Validatar automatically creates and refreshes helper views in the replication schema that simplify common reporting queries:

The VW_DATASET_TABLES and VW_DATASET_COLUMNS views are dynamically generated based on your custom field and profile definitions. When custom fields or profile definitions change, these views are automatically rebuilt on the next replication run.

Schema Evolution

Validatar handles schema changes automatically. On every replication run, the system compares the expected column structure of each Snowflake table against the actual table in Snowflake. If a mismatch is detected — a new column was added to the Validatar model, a column type changed, or a column was removed — the Snowflake table is dropped and recreated with the correct schema. The sync date for that table is reset, triggering a full reload on the next pass.

This means you don't need to manually manage the Snowflake schema as Validatar evolves. Upgrades that add new columns to internal tables are handled transparently.

Replication Status

After the initial replication completes, the Replication Status section on the configuration page displays per-table statistics:

• Table Name — the Snowflake table
• Row Count — total rows in the Snowflake table
• Unchanged — rows that haven't changed since the last sync
• Modified — rows updated since the last sync
• Inserted — new rows since the last sync

You can also view a comparison between your Validatar repository and Snowflake to identify any unexpected mismatches. If a table shows a discrepancy, you can clear its sync date to force a full re-sync.

Best Practices

• Schedule replication during off-peak hours — The initial full sync can be resource-intensive. Schedule it for a time when both your Validatar instance and Snowflake have available capacity.
• Use a dedicated schema — Keep replicated data separate from other Snowflake objects. The default REPOSITORY schema is a good choice.
• Monitor the Replication Status page — Check periodically for unexpected mismatches between Validatar and Snowflake row counts. Small differences immediately after a sync are normal (new data may have arrived between the sync and the comparison), but persistent large mismatches indicate an issue.
• Grant read-only access in Snowflake — Create a Snowflake role with SELECT-only permissions on the replication schema for your reporting and BI users. Don't modify the replicated tables directly — Validatar manages them.
• Use the helper views for reporting — The VW_DATASET_TABLES and VW_DATASET_COLUMNS views are designed for BI tools and dashboards. They pre-join the most commonly needed dimensions, saving you from writing complex queries against the raw tables.

How This Fits Into the Bigger Picture

Repository replication connects Validatar's internal data to your broader analytics ecosystem. With your metadata, test results, profiling metrics, and trust scores in Snowflake, you can:

• Build custom dashboards in tools like Tableau, Power BI, or Looker
• Run cross-environment comparisons by replicating multiple Validatar instances to the same Snowflake account
• Create automated alerts based on test result trends or trust score changes
• Feed Validatar data into data governance platforms for unified metadata management
• Perform historical analysis of testing coverage, execution trends, and data quality improvements over time

For more on data processing engines, see What Is a Data Processing Engine? and Creating Data Processing Engines.