How Trust Scores Are Calculated

This article covers the stored trust score: the score Validatar calculates for every table once a day and keeps as history. It explains which test runs count, how the quality, coverage and trust scores are worked out, how table scores roll up to schemas and data sources, and when the calculation runs.

The KPI bar above test lists and the Test Execution Activity panel in the Catalog calculate their own scores live, over a date range you pick. Those are covered in Live Quality Scores in Projects and the Catalog. To see which number on screen comes from where, start with Understanding Trust Scores.

The three stored scores

Trust Score = Quality Score × Coverage Score × 5

Each score is calculated per table. Schemas, data sources and the catalog show averages of the table scores (see Schema and data source scores).

Which test runs count

For each table, the calculation collects every test run that meets all of these conditions:

• The run started inside the table's look back period, 14 days by default. See When the calculation runs for the exact window.
• When the test ran, it was linked to the table, or to a column of the table, by a metadata link marked Impacts Trust Score.
• The data source, schema and table are active, and the table is included in trust scores. See Trust Score Settings.

Every run counts, not only the latest one. A test that ran 14 times in the window adds 14 runs to the score. A test that ran twice adds two. How often a test runs is therefore part of its weight: a daily test has seven times the influence of a weekly test with the same severity.

Which links impact the trust score

Each metadata link on a test carries an Impacts Trust Score setting. The icon beside the link shows its state, and its tooltip reads Impacts Trust Score or Does Not Impact Trust Score. By default, links on the test data set have the flag on and links on the control data set have it off. In a comparison test that reads from a source table and a target table, the result counts toward the table on the test side only. You can switch the setting for each link.

Validatar copies the links that impact the trust score onto each run when the run starts. Changing a link later does not change which tables earlier runs count toward.

Details that affect the count

• A run that references the same table more than once, for example through three of its columns, counts once for that table.
• The severity level and quality dimension come from the version of the test that ran. Changing a test's severity from Medium to High affects runs from then on, not runs already in the window.
• Severity weights are read from the current settings when the score is calculated. Changing the weight of the High level re-weights every High run in the window at the next calculation.
• The calculation does not check whether the test still exists. Runs of a deleted test keep counting until they age out of the window.

Quality Score

Quality Score = Σ (run score × severity weight) ÷ Σ (severity weight)

The sums cover every counted run for the table. The numerator is the weight the runs earned and the denominator is the weight they could have earned.

Severity weight

Each test has a severity level, and each level has a weight. The defaults are:

One Critical run carries the same weight as 16 Low runs. Informational runs have a weight of 0, so they never raise or lower the quality score (they still count toward coverage). Weights are set on Settings > Trust Scores > Severity Levels; see Severity Levels.

Run score

Each run gets its own score between 0 and 1. The test's Quality Score Method decides how:

With All or Nothing on every test, the formula becomes the one most people know: the sum of the severity weights of passing runs divided by the sum of the severity weights of all runs. With a proportional method, a failed run that compared 10,000 rows and found 50 bad ones still earns 0.995 of its weight.

A custom calculation can use the variables overallResult (Passed, Failed or Error), passedRowCount, failedRowCount, errorRowCount and totalRowCount.

Errors

A run that ended in error sits in the denominator at its full weight. Under All or Nothing it earns 0, so a connection failure lowers the quality score as much as a failed test does.

Coverage Score

The coverage score asks whether a table is tested from enough angles. It is worked out in two steps.

1. Count the distinct quality dimensions among the table's counted runs. The None dimension does not count. Pass or fail does not matter here, and Informational tests count: a dimension is covered as soon as any test in that dimension ran against the table in the window.
2. Look up the cumulative coverage score for that count in the Coverage Weighting table on Settings > Trust Scores > Settings.

0 dimensions                         → Coverage Score = 0
n dimensions, n below the table size → Coverage Score = cumulative weight for row n
n at or above the table size (5)     → Coverage Score = 1

The coverage weighting table has five rows. Each row adds a weight, and the cumulative column is the coverage score for that many dimensions.

Note: With the default weighting, any table with at least one tested dimension has a coverage score of 100%, so its trust score is simply its quality score on a five-star scale. A table whose tests all have the quality dimension None has a coverage score of 0 and a trust score of 0, however many tests pass. Assign a quality dimension to every test that should count.

Coverage weighting can be set globally and overridden for a data source, schema or table. The calculation uses the most specific setting that exists.

Worked example

The customer_orders table has five tests, all using All or Nothing, with a 14-day look back period and the example custom coverage weighting above.

Quality score

Earned   = 16×14 + 9×12 + 4×0 + 1×14 + 0×10 = 224 + 108 + 0 + 14 + 0 = 346
Possible = 16×14 + 9×14 + 4×2 + 1×14 + 0×14 = 224 + 126 + 8 + 14 + 0 = 372

Quality Score = 346 ÷ 372 = 0.930  (93%)

The Validity test failed every time, but it runs weekly, so its two runs cost only 8 of 372 possible points. The Informational test's four failures cost nothing.

Coverage score

Five dimensions ran: Completeness, Uniqueness, Validity, Consistency and Timeliness. Five reaches the end of the weighting table, so the coverage score is 100%. The Informational test supplies the fifth dimension even though it has no weight.

Trust score

Trust Score = 0.930 × 1.00 × 5 = 4.65

Two changes show how sensitive the score is to settings:

• Remove the Informational test and only four dimensions remain. Coverage drops to 95% and the trust score to 0.930 × 0.95 × 5 = 4.42.
• Switch the uniqueness test to Proportional. If each failed run found 50 duplicates in 10,000 rows, each failed run earns 9 × 0.995 = 8.955 instead of 0. The quality score rises to 363.91 ÷ 372 = 0.978 and the trust score to 4.89.

Tables with no runs in the window

A table that is included in trust scores but had no counted runs in the window still gets a record each day, with a quality score of 0, a coverage score of 0, a trust score of 0 and a target badge of Unknown. The REST API reports a test execution count of 0 for these records, which is how integrations tell "untested" apart from "tested and failing".

Schema and data source scores

A schema's or data source's trust score is the plain average of the current trust scores of its included tables. The average quality score and average coverage score are worked out the same way, as averages of the table percentages. They are not pooled from the underlying runs.

The averages are not weighted by row count, impact rating or number of tests. Tables with no runs in the window count as 0. A data source with 10 tables at 5.0 and 40 included but untested tables shows an average trust score of 1.0.

To separate "badly tested" from "not tested", use the counts beside the average:

To keep untested tables out of an average, exclude them from trust scores on the table, schema or data source (see Trust Score Settings).

When the calculation runs

Scheduled calculation

The Table Trust Score Calculation schedule runs once a day, by default one minute after midnight. Day boundaries use the Time Zone on Settings > Trust Scores > Settings.

A scheduled run covers the look back period up to midnight at the start of the run day. With a 14-day period, the run in the early hours of 10 March covers runs that started from 24 February through 9 March. The record is dated 10 March.

The scheduled run also marks tests whose latest run is older than the global look back period as Stale Passed, Stale Failed or Stale Errored, and raises an event for each change.

Manual refresh

Choose Refresh Trust Score on a table, or Refresh Trust Scores on a schema or data source, from the trust score menu at the top of the page. A manual refresh includes today: with a 14-day period on 10 March, it covers runs from 25 February through the moment of the refresh. It replaces today's record for the tables in scope.

A manual refresh does not mark tests stale.

History

Validatar keeps one record per table per day, so the trust score history shows how the score moved over time. If the score is recalculated on the same day, that day's record is replaced. The Trust Score Performance chart on the table, schema and data source pages draws from these records. Its 7, 14 and 30 day options change how much history is shown, not how scores are calculated. For a schema or data source, each point on the chart is the average of that day's table scores.

Where the stored scores appear

Related

• Understanding Trust Scores
• Trust Score Settings
• Quality Dimensions
• Severity Levels
• Impact Ratings
• Live Quality Scores in Projects and the Catalog