Data Quality Monitoring Project
  • 30 Sep 2022
  • 1 Minute to read
  • PDF

Data Quality Monitoring Project

  • PDF

Overview

Are you connecting to Snowflake data sources? Use the Data Quality Monitoring (DQM) quick start project to jumpstart your QA testing. Oftentimes, the most challenging part of data testing is knowing where to start. The DQM project aims to give you standard monitoring, formatting, and easy-to-use tests straight out of the box. To aid in the standardization of data quality testing, it is important to utilize and continuously monitor data profiling stats over time. 

You'll find several types of tests in the DQM project: 

  • Comprehensive Monitoring will detect any changes in data profiles such as record counts, duplicate values, null values, and more
  • You can set Expectations for data values and be notified if that value falls out of range or hasn't been updated. 
  • Field Format

How to Use the Quick Start Project

  1. Choose which Data Source to create the project from
  2. Using the Explorer, add DQ Monitoring custom field values

Quick start projects are used to test one data source.

Comprehensive Monitoring

Profiling your data not only presents the current state of your data but helps you to monitor changes over time. Tests in the Comprehensive Monitoring folder are standard table and column-level profiles that indicate deviations from your typical state of data quality. 

List of Column Tests

Test NameDescription
Blanks Showing Up in a Previously Non-Blank Column
Column Average Changed more than X Percent
Column Max Changed more than X Percent
Distinct Count Changed more than X Percent
Duplicates Showing Up in Previously Unique Column
Negative Showing Up in Previously Non-Negative Column
Nulls Showing Up in Previously Non-Null Column
Zero Showing Up in Previously Non-Zero Column


Custom Fields: Object Inclusion

The values you input for each data quality monitoring custom field drives how each test is configured. The custom fields are used to create the test name, filter which metadata objects the test can be run on, and the value success criteria for the test.

Schema Level

Choose how to enable DQM tests for the chosen schema. Tests included can be

  • All Table and Column Tests
  • All Table Level Tests Only
  • All Column Level Tests Only
  • Only Specific Tables and Columns
  • None

Expectations

Custom Fields: Object Expectations




Was this article helpful?