Creating Data Processing Engines

Overview

Validatar uses a data processing engine to compare the results of the two datasets executed within a standard test to evaluate whether the test should pass or fail. There are two types of data processing engines: Built-In and Snowflake.

Validatar includes a single built-in engine by default and supports adding one or more Snowflake engines. Each Snowflake engine corresponds to a separate Snowflake account.

Creating a Snowflake Engine

Navigate to the Settings > Data Processing Engines page to view the existing engines within your instance of Validatar. Click the "New Engine" button in the toolbar at the top to open a new page that allows you to enter a name for the new engine and complete the following three steps:

Step 1: Configure Connection

To complete this step, you will first need to retrieve your Snowflake account name by logging into your Snowflake console. Identify your Snowflake account URL by clicking the menu at the bottom left, then clicking the link icon to copy your account URL, which should look something like this: https://<locator>.<region>.snowflakecomputing.com. Enter the highlighted text as the Account Name in Validatar.

You can either use the predefined values for the remaining fields or customize as needed. Once you've completed this section, click the Save button at the top to create the data processing engine, then continue with the next step.

Step 2: Configure Snowflake

Once the engine has been created, you'll need to configure your Snowflake environment so that it can be used by Validatar.

• First click the "Generate Setup Script" button to open a pop-up with the initial setup script to create the database, schema, role, user, and warehouse objects if they do not already exist in your Snowflake account. Copy this script and paste it in your Snowflake console, then execute it using a role that has been granted access to the securityadmin and sysadmin system roles. NOTE: Make sure to replace the [[REPLACE]] placeholder text in the script with the value you want to use for the new user's password.
• Once you have run the setup script, click the "View Public Key/Script" button to open a pop-up with the script that attaches a public key to the new user that was created in the prior script. Copy this script and paste it in your Snowflake console, then execute it using a role that has been granted access to the securityadmin system role. Validatar authenticates to Snowflake using the private key stored securely within the Validatar repository database. 

Step 3: Validate Connection

Now that your Snowflake environment has been configured, the final step is to verify that Validatar can connect to it. Click the "Validate Settings" button to perform the steps listed below.

Once Validatar can successfully perform all of these steps, you'll be able to start using your new data processing engine by assigning data sources to it.