What this pack contains
36 tests for Power BI, built on the Power BI Service Data Source Template version 2.0. They check what is usually wrong when someone says a report is wrong: the data is stale, a refresh failed, a relationship drops rows into a blank member, a measure is broken, a visual points at a field that was renamed, the numbers disagree with the warehouse, a page renders blank, or row-level security shows a user rows they should not see.
The folders are named by what the tests need, so you can see at a glance which ones will run on your tenant:
| Folder | Tests | Needs |
|---|---|---|
| 1 Any license / Freshness | 7 | Any license |
| 1 Any license / Model integrity | 7 | Any license |
| 1 Any license / Change control | 8 | Any license; three need Contributor |
| 1 Any license / Reconciliation | 5 | Any license, plus a warehouse data source holding the same rows |
| 2 Capacity / Rendering | 6 | Fabric, Premium or Embedded capacity |
| 3 RLS | 3 | Contributor for two, capacity for one |
Each test's description starts with a Requires: line naming its tier. Testing Power BI: What Works on Which License explains the tiers.
Each top-level folder has a job, and one job runs them all:
| Job | Runs |
|---|---|
[Samples] Power BI - 1 Any license |
the 1 Any license folder |
[Samples] Power BI - 2 Capacity |
the 2 Capacity folder |
[Samples] Power BI - 3 RLS |
the 3 RLS folder |
[Samples] Power BI - ALL |
the three jobs above, as three Job steps |
Marketplace: Power BI Sample Tests
How the tests are built
Every test is a short Python script that calls the template's own helpers (resolve_model, relationship_orphans, measure_errors, report_missing_fields, visual_image and so on) and leaves a data frame for Validatar to judge. Five shapes recur:
- Count of problems. The script returns how many problems it found, and the test passes at zero. Every template test in the pack works this way.
- Detail. The script returns one row per problem with a LOCATION, a DETAIL and a VALUE. The control is an empty table, so every row is a failure. Integrity - Orphan keys in Fixture Sales (detail) lists each relationship with orphans and its sample keys.
- List. The script returns KEY and VALUE rows and the Control script returns the expected list: the approved refresh schedule, the report's pages, the approved role filters.
- Reconciliation. The Test side is DAX through the template; the Control side is SQL on the warehouse data source. Values must agree within 0.01%.
- Drift. The script returns numbers and the control is the average of the test's own last ten runs. The first run has no history and passes; it seeds the window.
Template tests fan out over the catalog
Twelve tests are template tests. Each one has a metadata filter on the catalog schema, such as SchemaName starts with Semantic Models:, and materializes one child test per matching catalog object. The child test's script receives the object through {{schema.name}} and {{table.name}}, so Integrity - No orphan keys on any relationship becomes one child test per semantic model in every workspace the data source ingests. Import the pack, run ingestion, materialize, and the template tests cover your own models with no editing. Child tests land in a subfolder named after their template test.
| Template test | Filter | One child per |
|---|---|---|
| Freshness - Every semantic model answers a query | Semantic Models: |
semantic model |
| Freshness - Import models refreshed successfully in the last 24 hours | Semantic Models: |
semantic model |
| Freshness - Latest refresh attempt did not fail | Semantic Models: |
semantic model |
| Freshness - Model parameters point at production | Semantic Models: |
semantic model |
| Integrity - No orphan keys on any relationship | Semantic Models: |
semantic model |
| Integrity - No duplicate keys on the one side of a relationship | Semantic Models: |
semantic model |
| Integrity - Every measure evaluates | Semantic Models: |
semantic model |
| Integrity - Dimension tables have no blank attributes | Model: |
model table |
| Change - Every visual's fields exist in its model | Reports: |
report |
| Change - Dashboard tiles point at items that exist | Dashboards: |
dashboard |
| Render - Every report page renders without Power BI errors | Reports: |
report |
| Reconcile - Model table row counts equal the warehouse | Model: Validatar Power BI Fixture / Fixture Sales, excluding (Measures) |
model table |
The reconciliation template is the exception that names one model, because the warehouse side has to hold the same tables. Change its filter to your model's schema.
The tests
The On the fixture column is the outcome on the fixture described below. Several tests fail there on purpose: the fixture carries planted defects so you can see what each check finds.
1 Any license / Freshness
| Test | Type | What it catches | On the fixture |
|---|---|---|---|
| Freshness - Every semantic model answers a query | Template | Expired DirectQuery credentials, a gateway that is down, a model that will not load. It counts one table's rows, choosing a DirectQuery table so the query reaches the source. Models with row-level security report RLS and pass | All three pass |
| Freshness - Import models refreshed successfully in the last 24 hours | Template | A model whose last successful refresh is older than MAX_HOURS, or that never refreshed. DirectQuery models pass |
Fixture Refresh Failure fails once its only successful refresh is a day old |
| Freshness - Latest refresh attempt did not fail | Template | A failed refresh, with Power BI's error code and message in the run log | Fixture Refresh Failure fails |
| Freshness - Model parameters point at production | Template | A model published while its Power Query parameters still point at development | Fixture Refresh Failure fails (Environment = Development) |
| Freshness - Last refresh took no more than 150% of the recent median | Standard, range | A refresh that suddenly takes much longer, usually a changed source query or gateway. Compares only once there are five earlier refreshes longer than five minutes | Passes |
| Freshness - Refresh schedule is as approved | Standard, list | A schedule switched off or moved | Passes |
| Freshness - Total Sales within 10% of recent runs | Standard, drift | A total that jumps or collapses between refreshes: a load that ran twice or not at all | Passes |
1 Any license / Model integrity
| Test | Type | What it catches | On the fixture |
|---|---|---|---|
| Integrity - No orphan keys on any relationship | Template | Fact rows whose key has no dimension row. They show as a blank member in every visual sliced by that dimension | Fixture Sales fails (3 rows with ProductKey 99); Fixture Sales RLS errors, because a service principal cannot query it |
| Integrity - No duplicate keys on the one side of a relationship | Template | Repeated keys on the side that should be unique, which double-count. Import models refuse them; DirectQuery and many-to-many relationships accept them | Fixture Sales fails (StoreKey 6); Fixture Sales RLS errors, because a service principal cannot query it |
| Integrity - Every measure evaluates | Template | Measures in an error state or that fail at the grand total | Fixture Sales fails (Legacy Margin, Fragile Ratio); Fixture Sales RLS errors, because a service principal cannot query it |
| Integrity - Dimension tables have no blank attributes | Template | Blank values in the visible text columns of tables on the one side of a relationship: blank slicer items and unlabelled bars | Product and Customer in Fixture Sales fail; the RLS model's dimension tables error, because a service principal cannot query them |
| Integrity - Model table row counts within 10 points of recent runs | Standard, drift | A table whose share of the model's rows moves out of step with the rest | Passes |
| Integrity - Orphan keys in Fixture Sales (detail) | Standard, detail | The relationship, row count and sample keys behind the orphan count | Fails with one row: Sales[ProductKey] -> Product[ProductKey], 3 rows, key 99 |
| Integrity - Measures that fail in Fixture Sales (detail) | Standard, detail | Each failing measure with Power BI's error message | Fails with Legacy Margin and Fragile Ratio |
1 Any license / Change control
| Test | Type | What it catches | On the fixture |
|---|---|---|---|
| Change - Model columns unchanged since recent runs | Standard, drift | A column retyped or removed. Added columns are listed in the run log | Passes |
| Change - Measure definitions unchanged since recent runs | Standard, drift (Contributor) | Any edit to a measure's DAX, by fingerprinting each expression from the model definition | Passes |
| Change - Relationships unchanged since recent runs | Standard, drift | A relationship deleted, deactivated, or switched to both directions | Passes |
| Change - Two models have the same columns and types | Standard, list (Contributor) | Development and production copies of a model that have drifted apart | Passes comparing Fixture Sales with Fixture Sales RLS, RegionManager ignored |
| Change - Every visual's fields exist in its model | Template (Contributor) | Visuals bound to a column or measure that was renamed or deleted: "Something's wrong with one or more fields" | Fixture Broken Report fails (Sales[Discontinued Measure], Product[Legacy Code]) |
| Change - Report is bound to the expected model | Standard, string | A report rebound to a test copy or the wrong model | Passes |
| Change - Expected report pages are present | Standard, list | A page deleted, renamed or reordered | Passes (Overview, Products, Trend) |
| Change - Dashboard tiles point at items that exist | Template | Tiles whose report or model is gone | Passes (the fixture dashboard has no tiles) |
1 Any license / Reconciliation
These need a warehouse data source holding the same rows as the model. They ship pointing at the fixture rows in the Snowflake schema DST_FACTORY_FIXTURES.POWERBI_FIXTURE; see The fixture.
| Test | Type | What it catches | On the fixture |
|---|---|---|---|
| Reconcile - Total Sales equals the warehouse | Standard | A measure total that differs from the same SUM in SQL | Equal: 234,210.00 |
| Reconcile - Total Sales by Region equals the warehouse GROUP BY | Standard | A measure grouped by a column that differs from the GROUP BY, region by region | Equal for all four regions |
| Reconcile - Distinct customers equal the warehouse COUNT DISTINCT | Standard | A distinct-count measure that differs from COUNT(DISTINCT) | Equal: 12 |
| Reconcile - A visual's own query equals the warehouse | Standard | The exact DAX a visual sends, copied from Performance Analyzer in Power BI Desktop, compared with SQL | Equal for every category |
| Reconcile - Model table row counts equal the warehouse | Template | A model table with more or fewer rows than its source table | Equal for every Fixture Sales table |
2 Capacity / Rendering
| Test | Type | What it catches | On the fixture |
|---|---|---|---|
| Render - Every report page renders without Power BI errors | Template | Pages that render blank or show "Can't display this visual", "Couldn't retrieve the data for this visual", "Something went wrong" or "See details". A report on a model with row-level security errors, because Power BI renders it only as a user | Fixture Broken Report fails; Fixture Regional Report errors |
| Render - Total Sales card shows the model's value | Standard | A card that shows a different number from the measure in the model. The card is read by its visual name | Equal: 234,210.00 |
| Render - Region-filtered card shows only that region | Standard | A report filter that does not reach the visual | East: 59,970.00 |
| Render - Overview page looks the same as recent runs | Standard, drift | A visual that went blank, moved or changed type, from luminance bands and a 4 x 4 grid | Passes |
| Render - Brand colour present on the Overview page | Standard, range | A report that lost its theme | Passes (#118DFF covers more than 1%) |
| Render - Required text is on the Overview page | Standard | A missing title, disclaimer or data-as-of label | Passes |
3 RLS
| Test | Type | What it catches | On the fixture |
|---|---|---|---|
| RLS - Role filters equal the approved expressions | Standard, list (Contributor) | A role added, removed or loosened | Passes |
| RLS - Every fact table is filtered by every role | Standard (Contributor) | A fact table a role's filters never reach, so its members see every row | Passes with Budget listed as an allowed exception; fails with the exception removed |
| RLS - Report rendered as the East manager shows only East | Standard (Capacity) | A role that renders the wrong rows, using Export To File with an effective identity | 59,970.00 |
Prerequisites
- A data source built from the Power BI Service Data Source Template 2.0, with metadata ingestion run and the Power BI - Catalog Custom Fields imported.
- The service principal as Contributor on the workspaces, for the tests marked Contributor.
- For the Rendering folder and the rendered RLS test: the workspace on Fabric, Premium or Embedded capacity, and a Data Agent with
pymupdf,pillow,pytesseractand the Tesseract binary. These tests error on shared or Premium Per User capacity. - For the Reconciliation folder: a warehouse data source with the same rows.
Import
- In Validatar, open Marketplace from the main navigation.
- Find Power BI Sample Tests and choose Import.
- Pick the target project. Validatar opens that project's import screen with the pack loaded.
- Map the pack's data sources: Power BI Service to your Power BI data source, and Warehouse (Snowflake) to the warehouse data source for the Reconciliation folder.
- Choose a target folder and commit.
- Materialize the template tests, so their child tests are created from your catalog.
Run the ALL job once to seed the drift tests, then a second time to see them compare.
To import from a downloaded file instead, take the pack XML from the marketplace listing, open Import in the target project, select the file, map the data sources and commit.
Adjusting a test
Every script opens with a PARAMETERS block, and nothing below it needs editing to retarget the test. Standard tests ship pointing at fixture objects:
# ============================ PARAMETERS ============================
# The workspace's Semantic Models schema in the catalog (or the bare workspace name).
WORKSPACE = 'Semantic Models: Validatar Power BI Fixture'
# The semantic model.
MODEL = 'Fixture Sales'
# Workspace of the model to compare with.
OTHER_WORKSPACE = 'Semantic Models: Validatar Power BI Fixture'
# The model to compare with (the Control side).
OTHER_MODEL = 'Fixture Sales RLS'
# Tables only one of the models is expected to have.
IGNORE_TABLES = ['RegionManager']
#
# Fails on: IGNORE_TABLES = []
# Set USE_FAILING_EXAMPLE = True to run this test against that failing example.
USE_FAILING_EXAMPLE = False
if USE_FAILING_EXAMPLE:
IGNORE_TABLES = []
# ====================================================================
- Point it at your model or report. Change
WORKSPACEandMODEL(orREPORT).WORKSPACEtakes the catalog schema name or a bare workspace name. - Change what it expects. Lists of approved values (pages, role filters, the refresh schedule) live in the Control script; thresholds such as
MAX_HOURSandEXPECTEDlive in the PARAMETERS block. - See it fail. Two tests carry a
Fails on:line and aUSE_FAILING_EXAMPLEswitch that points them at the fixture's failing case. The template tests need no switch: the fixture's planted defects already make specific child tests fail. - Template tests need nothing. Their PARAMETERS are filled in per child test. To narrow one, edit its metadata filter.
- Thresholds that decide pass or fail live on the Control tab: the drift tolerance, the reconciliation tolerance, the range a value must fall in.
The fixture
The pack ships pointing at the Validatar Power BI Fixture workspace, built from code so its numbers never move. It holds synthetic sales data: 240 order lines, January to March 2026, four regions, Total Sales 234,210.00.
| Item | What it is |
|---|---|
| Fixture Sales | Import model with Region, Product, Customer, Store, Date, Sales and Budget tables and nine measures. No row-level security |
| Fixture Sales RLS | The same tables plus RegionManager, with three roles: Region East, Region West and Regional Managers (dynamic, by user principal name) |
| Fixture Refresh Failure | One small table. Its first refresh succeeded and its latest refresh fails on purpose. Its Environment parameter says Development |
| Fixture Sales Report | Pages Overview, Products and Trend, with named visuals: kpiTotalSales, salesByRegion, salesByCategory and others |
| Fixture Broken Report | Pages Legacy KPIs and Still Fine. Visuals bound to a measure and a column that do not exist, and a card on a measure that raises an error |
| Fixture Regional Report | One page bound to Fixture Sales RLS, for rendering as a user |
| Fixture Dashboard | An empty dashboard |
The planted defects:
- Three sales rows carry ProductKey 99, which has no Product row (orphan keys).
- StoreKey 6 appears twice in Store, joined many-to-many (duplicate key).
- Product 8 has a blank Category and customers 11 and 12 have no Email (blank attributes).
- Measure Legacy Margin references a column that does not exist; Fragile Ratio raises an error.
- Budget has no relationship to Region, so the RLS roles do not filter it.
- Fixture Refresh Failure's latest refresh fails and its Environment parameter is Development.
The Reconciliation folder's Control side reads the same rows from Snowflake, in tables and columns named exactly like the model's ("Sales", "Amount"). Against your own warehouse, edit the Control SQL for your naming.
To use the pack on your own tenant, keep the template tests as they are, and retarget the standard tests' PARAMETERS blocks at your models and reports.
Troubleshooting
Import reports an unknown data source. Map Power BI Service and Warehouse (Snowflake) to your data sources during import.
Template tests have no child tests. Run metadata ingestion on the Power BI data source, then materialize the template tests. A template test's filter matches catalog schema names, so it finds nothing before ingestion.
Child tests for a model with row-level security error. A service principal cannot run DAX against those models. Narrow the template test's filter to leave them out, and cover them with the role-definition and rendering tests in the 3 RLS folder.
Rendering tests error with a 403 that mentions capacity. The workspace is on shared or Premium Per User capacity. Move it to Fabric, Premium or Embedded capacity.
Reconciliation tests error with a missing table. The warehouse data source does not hold the fixture rows, or the Control SQL names a schema you do not have. Retarget the Control SQL.
Drift tests pass on the first run no matter what. By design. Their control is the average of the test's previous runs, and Control Missing Result Action is set to Pass so the first run seeds history.