Create and Run a Profile Set

Prev Next

To profile a data source, open it in Settings > Data Sources, go to Profiling > Profile Sets, click New, choose the profiles and the tables to profile, and save. Then click Run to profile now, or Set Schedule to profile on a schedule.

Applies to: Validatar Cloud · Validatar Server

A profile set is a saved selection of data profiles and objects on one data source. Each run records row counts, null counts, distinct counts, ranges, most common values and the other profiles you chose, for every table and column the set covers. The values show up in the Catalog, build up a history you can watch, and can feed tests that fail when a value moves.

The example on this page is Star Schema Profile, a profile set on the Enterprise Data Warehouse (Snowflake). It runs 19 profiles over every table in STAR_SCHEMA: volumes, nulls, distinct values, ranges and the most common values.

Before you start

  • The data source has a data source template and its schema metadata has been ingested. Without both, Profile Sets shows "Before a data source can be profiled, the connection used for profiling needs to be assigned to a Data Source Template" instead of the list. See Create a Data Source and Ingesting Schema Metadata.
  • Manage Data Profiling permission on the data source to create and change profile sets, and Run Profile Sets to run them. Data Source Admins and the data source owner have both.
  • Set Schedule needs a license that includes scheduling.

Create the profile set

  1. Go to Settings > Data Sources and open the data source.
  2. In the side panel, click Profiling > Profile Sets.
  3. Click New.
  4. Enter a Name, such as Star Schema Profile, and a Description of what the set covers and when it runs.
  5. On the Profile Selection tab, choose a starting group from Default Profile Sets, or check profiles one by one under Select Profiles.
  6. If Prerequisites lists a profile in orange, click the plus icon beside Prerequisites to add every missing one. Distinct Percent, for example, needs Distinct Count and Record Count.
  7. On the Object Selection tab, click Add Filter, choose the Field, the operator and the value, and click Add Filter in the dialog. For Star Schema Profile: Schema Name equals STAR_SCHEMA.
  8. Click Preview Objects to see the tables and columns the filters select.
  9. On the Profile Set Configuration tab, check the execution settings (table below).
  10. Click Save.

The Statistics tab beside the name updates as you change profiles and filters, so you can see how much work a run will be before you start it.

Default Profile Sets

The groups under Default Profile Sets are starting points. Pick one, then add or remove profiles; the group shows Custom once your selection no longer matches a group.

Group What it selects
Table Profile Record Count, Total Data MB and Column Count
Quick Profile The table profiles, plus distinct and null counts, the minimum and maximum of numbers, text and dates, and the mean of numbers
Standard Profile Quick Profile, plus standard deviation, text lengths, blanks, numeric text, zeros and negatives
Full Profile Every standard profile except Sample, including medians, quartiles, most common values and distributions
String Analysis Text columns: minimum and maximum, lengths, blanks, numeric text, longest and shortest values, most common values
Date Analysis Date columns: minimum, maximum, and year and year-month distributions
Numeric Analysis Number columns: minimum, maximum, mean, median, quartiles, standard deviation, zeros, negatives and bins
Distributions Distributions, top and bottom 10 values, bins, and year and year-month distributions

The profiles on offer come from the data source's template. 40 Standard Data Profiles describes each one, and Profiling Configuration SQL Templates shows how a template defines them.

Object Selection filters

Filters decide which tables and columns get profiled. Several filters narrow the set together. Useful fields include Schema Name, Table Name, Table Type, Table Record Count, Column Name, Column Data Type, Column Mapped Type, Column Nullable, Column Primary Key and Column Max Length.

A filter on Table Record Count keeps very large tables out of a set that runs often. Filters are applied each time the set runs, so a table added to STAR_SCHEMA next month is profiled without anyone editing the set, once it has been ingested.

Profile Set Configuration

Setting What it does
Profile Connection Primary Connection, or Profile Connection when the data source has a separate connection for profiling (set on the Profiling page)
Max Concurrent Queries How many profiling queries run at once. Enter 0 for the maximum your instance allows. Lower it to go easier on a busy source.
Max Profiles Per Query How many profiles Validatar combines into one query for an object. Higher means fewer queries.
Purge results after N days Deletes this set's profile results older than N days

Star Schema Profile uses 4 concurrent queries and up to 500 profiles per query.

On a Python data source, Profile Selection also asks for a Profile Script, and the concurrency setting is Max Concurrent Scripts.

Run it now or on a schedule

  1. Select the profile set on the Profile Sets list, or open it.
  2. Click Run and confirm. Validatar starts the run in the background.
  3. To profile on a schedule instead, click Set Schedule, set the recurrence and save. The Schedule tab on the profile set and the Next Run column on the list show when it runs next.

You can also add a profile set to a job as a step, so the profiles refresh in the same run as the tests that read them. See Create a Job.

See the results

  • Profile Executions, under Profiling in the data source's side panel, lists every run with its status, start and end dates, steps and errors. Select a run and click Results to see each table and column it profiled.
  • In the Catalog, a column's Column Profile tab shows its current values (counts, ranges, most common values, distributions) and the Profile History tab shows how they changed from run to run. Tables and data sources have a Profile History tab as well.
  • The Last Run column on the Profile Sets list shows when each set last ran.

Profile a few objects from the Catalog

For a one-time look at a schema, table or column, profile it straight from the Catalog (2024.3 and later). You need Run Profile Sets on the data source.

  1. In the Catalog, open a data source, schema or table.
  2. On its Schemas, Tables or Columns tab, select one row, or none to cover everything on the tab.
  3. Click Run Profiles.
  4. Choose the profiles in the Run Profiles dialog and click Run.

Nothing is saved as a profile set, and the values land in the same Catalog tabs.

Conditional profiles

A profile can carry a Profile Condition, set on the profile in its data source template or on a custom profile. The built-in templates use conditions so that expensive profiles run only where they mean something. Distribution (String), for example, runs only on columns with 1 to 100 distinct values that are not unique, and Null Count only on nullable columns.

From 2026.1.1, a condition that reads another profile's value is checked for each table or column during the run, so only the objects that meet it are profiled. The step detail of a profile execution shows the values used for each object, the condition as it was run, and which objects were skipped.

Do it with Validatar AI

Do it with Validatar AI (Validatar Cloud, beta). Open the AI panel and ask: "Add a profile set named Sales Schema Profile to the Wholesale Distribution System that profiles every table in the Sales schema with Record Count, Null Count, Null Percent, Distinct Count and Distinct Percent." Review what it creates before you run it. You can undo the change from the chat history.

The AI adds or updates a profile set as part of updating the data source. It cannot run or schedule a profile set, so click Run or Set Schedule yourself.

Troubleshooting

Symptom Cause Fix
Profile Sets shows a message instead of the list The connection has no data source template, or metadata was never ingested Assign a Data Source Template on the Connection page, then click Refresh Now on Schema Metadata.
New or Run is greyed out You lack Manage Data Profiling or Run Profile Sets on the data source Ask the data source owner or a Data Source Admin for the permission.
Set Schedule is greyed out No profile set is selected, or the license does not include scheduling Select one profile set. Ask your Validatar account team about scheduling.
A column has no value for a profile The profile does not apply to that data type, or its condition was not met Check the profile's data types on the Data Profiles page and the step detail of the run.
A run shows errors A query failed on the source, often permissions or a timeout Click the # Errors count on Profile Executions to read the errors.
Runs take too long or load the source Too many objects or profiles, or high concurrency Add filters such as Table Record Count, drop heavy profiles, or lower Max Concurrent Queries.
New tables are not profiled They are not in the catalog yet Ingest the schema metadata again, or set a refresh schedule on Schema Metadata.

Related

Engineers who script their setup can add profile sets through the Validatar MCP server and the Validatar Agent Kits.