Overview
Validatar uses a data processing engine to compare the results of the two datasets executed within a standard test to evaluate whether the test should pass or fail. There are three types of data processing engines: Built-In, Snowflake, and Databricks.
Validatar includes a single built-in engine by default and supports adding one or more Snowflake and Databricks engines. Each Snowflake engine corresponds to a separate Snowflake account, and each Databricks engine corresponds to a separate Databricks workspace. For more on how each engine type works, see What is a Data Processing Engine?
Creating an Engine
Navigate to the Settings > Data Processing Engines page to view the existing engines within your instance of Validatar. Click the New Engine button in the toolbar at the top, enter a name for the new engine, and choose its type. The setup steps depend on the engine type:
- Creating a Snowflake Data Processing Engine — key pair authentication, the Snowflake setup scripts, and connection validation.
- Creating a Databricks Data Processing Engine — service principal setup, workspace and warehouse permissions, the setup script, and connection validation.
Once an engine has been created and validated, you can start using it by assigning data sources to it.