Live Quality Scores in Projects and the Catalog

Two places in Validatar calculate a quality score on the spot, every time the page loads, from the test runs inside a date range you choose:

• The KPI bar at the top of a test folder in Projects and at the top of the Tests tab in the Catalog.
• The Test Execution Activity panel on the Info tab of a data source, schema, table or column in the Catalog.

Neither one is stored, and neither one feeds the stored trust score. Their date ranges are separate from the trust score look back period. This article explains how each is calculated and why their numbers can differ from the stored quality score described in How Trust Scores Are Calculated.

The two live calculations side by side

Validatar remembers the range you last picked on each page in your browser, so two people looking at the same folder can see different numbers.

KPI bar

The KPI bar has four modes: Summary, Current Status, Severity and Quality Dimension. On the Projects side you can filter by Data Source. On the Catalog side you can filter by Project.

Which tests and runs it includes

• In Projects: active tests in the folder and all of its subfolders. Job folders do not have a KPI bar.
• In the Catalog: active tests in projects you have access to whose current version is linked to the object, on either data set. A link does not need Impacts Trust Score to count here, so a comparison test appears on both its source table and its target table. Because the match uses the test's current links, a test that is relinked brings its earlier runs with it.
• Runs: every completed run in the range. A test that ran ten times counts ten times. Runs still in progress are left out.

Summary mode

Quality Score = Σ severity weight of passed runs ÷ Σ severity weight of all completed runs

• Failed runs and errored runs both earn 0 and both count in the denominator.
• The test's Quality Score Method is not used. A failed run earns nothing, even if 9,995 of its 10,000 rows passed.
• Informational runs have a weight of 0. They change neither side of the percentage but do count in Total Executions.
• Severity weights come from Settings > Trust Scores > Severity Levels as they are now, so changing a weight changes the score for past runs too.

The percentage is shown as a whole number rounded toward 50%: 99.6% shows as 99% and 0.3% shows as 1%, so a score is never shown as perfect or empty when it is not. Hover over the score to see the earned and possible weight.

The other Summary tiles count Total Executions, Standard Tests and Template Tests (Projects only), and New Tests, Modified Tests and Test Authors within the range.

Example. A folder holds a Critical test that ran 14 times and passed 13 times, and a Low test that ran 14 times and failed every time.

Earned   = 16×13 + 1×0  = 208
Possible = 16×14 + 1×14 = 238
Quality Score = 208 ÷ 238 = 87.4%, shown as 87%

Severity mode

One tile per severity level. Each tile shows the share of runs at that level that passed:

Severity tile = passed runs ÷ all completed runs at that severity

This is a plain count. Weights do not apply inside a single severity. The bar under each tile splits the runs into pass, fail and error. The first five levels are shown and any others are grouped under Other. A Quality Dimension filter narrows the runs.

Quality Dimension mode

One tile per quality dimension that had runs. Runs of tests with the None dimension are left out. Each tile shows the share of runs in that dimension that passed, and the bar shows the pass, fail and error split. A Severity Level filter narrows the runs. The first five dimensions are shown and the rest are grouped under Other.

Note: Each quality dimension carries a built-in weight that is not shown in settings. In this mode, a dimension whose weight is 0 shows 0% whatever its results. That applies to Standards, Relationships, Security, Precision, Performance, Processing, Compliance and any dimension you add. The Other tile adds run counts to weighted totals, so its percentage is not a pass rate. For these tiles, read the bar below the tile, which shows the true pass, fail and error split.

Current Status mode

Current Status ignores the date range. It counts tests, not runs, by each test's current overall status: Passing, Failing, Erroring, Intermittent, Stale, Not Run, Newly Failing and Newly Erroring. The Current Test Status card in the Catalog shows the same counts, without Newly Failing and Newly Erroring.

Test Execution Activity panel

The panel sits on the Info tab of a data source, schema, table or column. Its filters are Date Range, Severity, Quality Dimension and Metric.

Weighted Score

Weighted Score = Σ (run quality score × severity weight)  /  Σ (severity weight)

This is the same formula as the stored quality score, with three differences: it covers the date range you picked, it uses the severity weight recorded when each run started, and it is shown as two numbers rather than a percentage. Divide the first by the second to compare it with a quality score.

• The run quality score follows the test's Quality Score Method, so proportional tests earn partial credit. See How Trust Scores Are Calculated.
• Runs are matched to the object through the links that impact the trust score, as they were when the run started.
• Total Executions counts the runs, including Informational ones.
• A run that has started but not finished adds its weight to the second number and nothing to the first until it completes.
• The panel does not check project access or whether a test still exists, so runs from projects you cannot open and from deleted tests are included.

Note: When a test is linked to several columns of the same table, the panel counts its weight once for each link on the table, schema and data source pages, while Total Executions counts the run once. The stored quality score counts the run once. Tests with many column links therefore carry more weight in this panel than in the trust score.

Date range

Last 7 Days means today and the six days before it, in whole days. The same Date Range selection also sets how much history the Trust Score Performance chart shows on a table page. It does not change how the stored scores are calculated.

Metric

The Quality Dimension table uses severity weights. Quality dimension weights are not used anywhere in this panel.

Schemas and data sources

On a schema or data source, the panel pools every run linked to any table in scope and sums them. It does not average table scores. A schema with one heavily tested table and twenty untested ones shows the busy table's results, while the stored average trust score for the same schema counts the twenty untested tables as 0.

Why live and stored numbers differ

Expect the live scores and the stored quality score to differ. The usual reasons:

To audit one table, refresh its trust score by hand, then open its Test Execution Activity panel with the date range set to the same number of days as the look back period. A manual refresh and the panel both start their window at midnight that many days back, counting today, so the two use the same runs. Divide the panel's first number by its second and compare it with the quality score. Any remaining gap comes from a changed severity weight or from tests linked to several columns of the table.

Related

• Understanding Trust Scores
• How Trust Scores Are Calculated
• Severity Levels
• Quality Dimensions