Overview
Validatar uses a data processing engine to compare the results of the two datasets executed within a standard test to evaluate whether the test should pass or fail. There are three types of data processing engines: Built-In, Snowflake, and Databricks.
Each data source is assigned a data processing engine when setting up the primary connection to the database. When a standard test is executed, the data processing engine assigned to the data source selected in the "What do you want to test?" section is used to compare the results of both datasets.
Built-In Engine
The Built-In engine is a licensed feature. When this engine is used, Validatar retrieves the data from both datasets and processes it in memory on the Validatar web server. When the "Scripts Include Order By" option is enabled for a test, the data from both datasets is streamed to the web server instead of being fully loaded into memory, which greatly reduces the total amount of memory used by that test on the Validatar web server.
Results calculated from the built-in engine can be stored locally in the Validatar metadata repository, in Snowflake, or in Databricks.
Snowflake Engine
The Snowflake engine is a licensed feature. When this engine is used, data from both datasets is processed directly within the corresponding Snowflake account and only the summary result of the comparison is sent to the Validatar web server. Data retrieved using a data agent from external databases outside of Snowflake is pushed from the data agent directly into Snowflake.
Results calculated from the Snowflake engine are always stored in Snowflake.
To set one up, see Creating a Snowflake Data Processing Engine.
Databricks Engine
The Databricks engine is a licensed feature, available in Validatar 2026.1 and later. When this engine is used, data from both datasets is processed directly within your Databricks environment using a Databricks SQL warehouse, and only the summary result of the comparison is sent to the Validatar web server. Data retrieved using a data agent from external databases outside of Databricks is pushed from the data agent directly into temporary tables in Databricks.
Results calculated from the Databricks engine are always stored in Databricks, in a catalog and schema that you control.
To set one up, see Creating a Databricks Data Processing Engine.