How does Validatar Connect to Data?

Prev Next

Overview

Validatar connects to your data where it already lives: cloud warehouses, databases in your own data center, file stores, SaaS applications and BI reports. You pick the connection that fits each source. Once it is connected, a source works like every other one in Validatar. Its tables appear in the catalog, it can be profiled, and the same tests, templates and trust scores apply.

This page is the quick tour. Each section links to the full article.

At a Glance

Your data lives in Validatar connects through
Snowflake or Databricks A native connection, with the option to run every comparison inside your own account
SQL Server, PostgreSQL, Oracle, MySQL and other databases A built-in connection, or any ODBC or OLE DB driver
Salesforce, NetSuite, MongoDB, Workday and many more The CData Connector Library (228 sources)
REST APIs, SaaS tools and in-house systems A Python data source
Amazon S3, Azure Blob Storage, SFTP or a Windows share File templates from the Validatar Marketplace
Power BI The Power BI Service template
A private network The Validatar Data Agent, installed next to the data

Keep the Work Inside Snowflake or Databricks

Many Validatar tests compare two sets of data: a source and its target, or an old platform and its replacement. With a Snowflake or Databricks data processing engine, that comparison runs inside your own Snowflake account or Databricks workspace, on your compute. Only the summary result goes back to Validatar, and the detailed results stay in a database you control.

With the Snowflake engine, data the Data Agent pulls from a system outside Snowflake, such as an on-premises Oracle table, goes straight into your Snowflake account. The work still happens on your platform.

The same engine can hold a copy of Validatar's own data: the catalog, test results, profiling and trust scores, ready to query in SQL or chart in the BI tool your team already uses.

Both engines are licensed options. Without one, Validatar compares the data on its own server. What is a Data Processing Engine? explains the choices.

Reach Data Behind Your Firewall

Some databases should never face the internet. The Validatar Data Agent is a lightweight service you install inside your network, on Windows, Linux or Kubernetes. It runs queries against your sources from inside the network and sends the results to Validatar, so those databases can be tested where they sit. Several agents can work together as a group.

Learn more: Validatar Data Agent

SQL or Python: Two Ways In

SQL data sources are for anything that speaks SQL: relational databases, cloud warehouses and lakehouses. Validatar reads the schemas and columns, profiles the data and offers ready-made checks, so most tests need no hand-written SQL.

Python data sources cover everything else. A Python script reaches a REST API, a SaaS application, a file store or an in-house system and hands the data to Validatar as a table. If Python can read it, Validatar can test it.

Either way, the results look the same. A check on a Jira project reports just like a check on a Snowflake table.

Any Database With an ODBC Driver

If a database has an ODBC driver, Validatar can test it. Install the driver where Validatar runs its queries (on Validatar Server or a Data Agent), add a connection string, and the database gets its own catalog and profiling. Netezza and Amazon Athena both connect this way. SQL Server, PostgreSQL, Snowflake and Databricks also have built-in connection types, and OLE DB is supported as well.

Learn more: Connection Strings

228 Sources Through CData

Validatar has partnered with CData Software to bring CData's library of 228 connectors to Validatar Cloud and Validatar Server. It reaches well past the usual databases:

  • Databases such as Teradata, Db2 and SAP HANA
  • NoSQL and streaming stores such as MongoDB, Cosmos DB and Kafka
  • Business applications such as Salesforce, Dynamics 365, NetSuite, Workday, ServiceNow, HubSpot, Zendesk, SharePoint and Shopify

Where a connector supports it, Validatar sets the connection to read-only. Search the CData Connector Library by source, table or sign-in method.

Prebuilt Templates and Macros

A data source template is a ready-made setup for one platform. It knows how to read the platform's schemas, how its data types map, what to profile, and which common checks to offer. Choose the template when you connect, and the source is ready to use.

Templates for Snowflake, SQL Server, PostgreSQL and Databricks come built in. More are on the Validatar Marketplace: open Marketplace inside Validatar, choose a template, click Get, then Import Directly.

  • Databases and warehouses: Oracle, MySQL, Amazon Redshift, Teradata, DB2 and Amazon Athena
  • Files: AWS S3, Azure Blob Storage and Windows Directory
  • Applications: Salesforce and Jira Cloud
  • BI and catalogs: Power BI Service, Alation, Atlan and Collibra

Macros are where templates save the most time. A macro is a ready-made check with blanks for the schema, table and column, and Validatar fills those blanks with dropdowns of what actually exists in the source. For a platform that is not on the list, build a template once and import it into each of your Validatar environments.

Learn more: What is a Data Source Template? and Data Source Template Macros

Files, Reports and Catalogs

Files. The S3, Azure Blob Storage and Windows Directory templates read CSV, Excel, Parquet, Avro, JSON and XML files as tables. PDFs, images and text files can be tested too, from a required footer on every PDF page to the text on a scanned receipt. See AWS S3 Unstructured File Sample Tests.

Power BI. The numbers people act on usually reach them through a report, so Validatar tests the report as well as the warehouse under it. The Power BI Service Data Source Template catalogs workspaces, semantic models, reports and dashboards, and its checks catch stale data, totals that disagree with the warehouse, and visuals bound to fields that no longer exist.

Data catalogs. Alation, Atlan and Collibra connect in both directions. Owners, tags, glossary terms and lineage come into Validatar. Test results and trust scores go back to the catalog, so people browsing it can see whether the data passed its checks. See Alation Integration, Atlan Integration and Collibra Integration.

Validatar Cloud or Validatar Server

Validatar Cloud connects directly to sources it can reach over the internet, such as a cloud warehouse. Sources inside your network connect through a Data Agent.

Validatar Server runs inside your network and connects directly to the databases it can reach. A Data Agent covers segregated networks the server cannot reach.

Learn more: Cloud vs Server Comparison

See It on Your Own Sources

To see Validatar connect to your systems, contact us for a demo.