VW_DATASET_COLUMNS

Overview

VW_DATASET_COLUMNS provides detailed information about every column across all metadata tables in your Validatar environment. Each row represents one column and includes its data type, position, custom field values, and the full suite of column-level profile metrics (null counts, distinct counts, min/max values, distribution stats, and more).

This view is essential for data quality reporting, profiling dashboards, and column-level governance.

Schema

Identity Columns

Column	Type	Description
COLUMN_ID	NUMBER	Unique identifier for the column
COLUMN_NAME	VARCHAR	Name of the column
COLUMN_KEY	VARCHAR	Composite key in format `c.<COLUMN_ID>`
TABLE_ID	NUMBER	ID of the parent table
TABLE_NAME	VARCHAR	Name of the parent table
TABLE_KEY	VARCHAR	Parent table key (`t.<TABLE_ID>`)
TABLE_TYPE_ID	NUMBER	Type code of the parent table
TABLE_TYPE_NAME	VARCHAR	"Table" or "View"
SCHEMA_ID	NUMBER	ID of the schema
SCHEMA_NAME	VARCHAR	Name of the schema
DATA_SOURCE_ID	NUMBER	ID of the data source
DATA_SOURCE_NAME	VARCHAR	Name of the data source
FULL_PATH	VARCHAR	Fully qualified: `<schema>.<table>.<column>`

Column Metadata

Column	Type	Description
SEQUENCE	NUMBER	Ordinal position of the column in the table
DATA_TYPE	VARCHAR	Native data type as reported by the source system
MAPPED_TYPE_ID	NUMBER	Validatar's internal mapped type ID
MAPPED_TYPE_NAME	VARCHAR	Validatar's mapped type name (e.g., "Integer", "String", "Date")
IS_NULLABLE	BOOLEAN	Whether the column allows NULLs
IS_IDENTITY	BOOLEAN	Whether the column is an identity/auto-increment
IS_PRIMARY_KEY	BOOLEAN	Whether the column is part of the primary key
STRING_MAX_LENGTH	NUMBER	Maximum string length (string types only)
NUMERIC_PRECISION	NUMBER	Numeric precision (numeric types only)
NUMERIC_SCALE	NUMBER	Numeric scale (numeric types only)
DATE_TIME_PRECISION	NUMBER	Datetime precision
COMMENT	VARCHAR	Column comment from the source system

Date Columns

Column	Type	Description
DATE_FIRST_SEEN	TIMESTAMP	When Validatar first discovered this column
DATE_MODIFIED	TIMESTAMP	Last modification date
DATE_FIRST_PROFILED	TIMESTAMP	First profiling execution
DATE_LAST_PROFILED	TIMESTAMP	Most recent profiling execution
DATE_LAST_REFRESHED	TIMESTAMP	Last schema metadata refresh

Custom Field Columns

Common custom columns (varies by environment):

Column	Description
CUSTOM_FRIENDLY_NAME	User-assigned friendly name
CUSTOM_DESCRIPTION	User-assigned description
CUSTOM_COLUMN_TAGS	Comma-separated tags
CUSTOM_COLUMN_ROLE	Business role (e.g., "Business Key", "Surrogate Key", "Measure")
CUSTOM_FOREIGN_KEY_REFERENCE	Foreign key reference notation
CUSTOM_SENSITIVITY_LEVEL	Data sensitivity classification
CUSTOM_FIELD_FORMAT	Expected format pattern
CUSTOM_ENABLE_COLUMN_DQM_TESTS	Whether DQM tests are enabled
CUSTOM_PII_DETECTION_TAGS	PII detection results

Profile Metrics — Counts and Percentages

Column	Type	Description
PROFILE_DISTINCT_COUNT	DECIMAL	Number of distinct values
PROFILE_DISTINCT_PERCENT	DECIMAL	Distinct values as percentage of total rows
PROFILE_NULL_COUNT	DECIMAL	Number of NULL values
PROFILE_NULL_PERCENT	DECIMAL	NULL values as percentage of total rows
PROFILE_BLANK_COUNT	DECIMAL	Number of blank/empty string values
PROFILE_BLANK_PERCENT	DECIMAL	Blank values as percentage
PROFILE_NUMERIC_COUNT	DECIMAL	Count of values that are numeric
PROFILE_NUMERIC_PERCENT	DECIMAL	Numeric values as percentage
PROFILE_ZERO_COUNT	DECIMAL	Count of zero values
PROFILE_ZERO_PERCENT	DECIMAL	Zero values as percentage
PROFILE_NEGATIVE_COUNT	DECIMAL	Count of negative values
PROFILE_NEGATIVE_PERCENT	DECIMAL	Negative values as percentage

Profile Metrics — Value Statistics

Column	Type	Description
PROFILE_MOST_COMMON_VALUE	VARCHAR	Most frequently occurring value
PROFILE_MOST_COMMON_COUNT	DECIMAL	Frequency of the most common value
PROFILE_MIN_NUMERIC	DECIMAL	Minimum numeric value
PROFILE_MAX_NUMERIC	DECIMAL	Maximum numeric value
PROFILE_MEAN_NUMERIC	DECIMAL	Mean (average) numeric value
PROFILE_MEDIAN_NUMERIC	DECIMAL	Median numeric value
PROFILE_LOWER_QUARTILE	DECIMAL	25th percentile value
PROFILE_UPPER_QUARTILE	DECIMAL	75th percentile value
PROFILE_STANDARD_DEVIATION	DECIMAL	Standard deviation
PROFILE_STDDEV_LARGE	DECIMAL	Population standard deviation
PROFILE_MIN_STRING	VARCHAR	Minimum string value (alphabetical)
PROFILE_MAX_STRING	VARCHAR	Maximum string value (alphabetical)
PROFILE_MIN_LENGTH	DECIMAL	Shortest string length
PROFILE_MAX_LENGTH	DECIMAL	Longest string length
PROFILE_MEAN_LENGTH	DECIMAL	Average string length
PROFILE_MIN_DATE	TIMESTAMP	Earliest date value
PROFILE_MAX_DATE	TIMESTAMP	Latest date value
PROFILE_LONGEST_VALUE	VARCHAR	The longest actual value
PROFILE_SHORTEST_VALUE	VARCHAR	The shortest actual value

Profile Metrics — Checksums

Column	Type	Description
PROFILE_SUM_CHECKSUM_COLUMN	DECIMAL	Sum-based checksum for change detection
PROFILE_SUM_DISTINCT_CHECKSUM_COLUMN	DECIMAL	Distinct-value checksum

Generated Profile Display Columns

Column	Type	Description
GEN_PROFILE_MIN_VALUE	VARCHAR	Coalesced min value (numeric, string, or date)
GEN_PROFILE_MAX_VALUE	VARCHAR	Coalesced max value (numeric, string, or date)
GEN_PROFILE_NULL_DISPLAY	VARCHAR	Formatted null display: `<count> (<percent>%)`
GEN_PROFILE_DISTINCT_DISPLAY	VARCHAR	Formatted distinct display: `<count> (<percent>%)`
GEN_PROFILE_MOST_COMMON_DISPLAY	VARCHAR	Formatted: `<value> (<count>)`

Relationships

VW_DATASET_COLUMNS
    │
    ├── TABLE_ID → VW_DATASET_TABLES.TABLE_ID (many-to-one)
    ├── COLUMN_KEY → VW_DATASET_TESTS.TEST_PRIMARY_METADATA_LINK_OBJECT_KEY
    ├── COLUMN_KEY → VW_DATASET_METADATA_OBJECTS.OBJECT_KEY
    └── DATA_SOURCE_ID → VW_SECURE_USER_DATA_SOURCE_ACCESS.META_DATA_SOURCE_ID

Example Queries

Find Columns with High Null Percentages

SELECT
    DATA_SOURCE_NAME,
    FULL_PATH,
    PROFILE_NULL_PERCENT,
    PROFILE_NULL_COUNT,
    IS_NULLABLE
FROM REPOSITORY.VW_DATASET_COLUMNS
WHERE PROFILE_NULL_PERCENT > 50
ORDER BY PROFILE_NULL_PERCENT DESC;

Column Profile Summary for a Specific Table

SELECT
    COLUMN_NAME,
    DATA_TYPE,
    MAPPED_TYPE_NAME,
    GEN_PROFILE_MIN_VALUE,
    GEN_PROFILE_MAX_VALUE,
    GEN_PROFILE_NULL_DISPLAY,
    GEN_PROFILE_DISTINCT_DISPLAY,
    GEN_PROFILE_MOST_COMMON_DISPLAY
FROM REPOSITORY.VW_DATASET_COLUMNS
WHERE TABLE_NAME = 'DIM_CUSTOMER'
    AND SCHEMA_NAME = 'PUBLIC'
ORDER BY SEQUENCE;

Columns Where Values Exceed Expectations

SELECT
    FULL_PATH,
    PROFILE_MAX_NUMERIC,
    CUSTOM_EXPECTATION_MAXIMUM_MAX_VALUE,
    PROFILE_MIN_NUMERIC,
    CUSTOM_EXPECTATION_MINIMUM_MIN_VALUE
FROM REPOSITORY.VW_DATASET_COLUMNS
WHERE (CUSTOM_EXPECTATION_MAXIMUM_MAX_VALUE IS NOT NULL
       AND PROFILE_MAX_NUMERIC > TRY_CAST(CUSTOM_EXPECTATION_MAXIMUM_MAX_VALUE AS DECIMAL(38,8)))
   OR (CUSTOM_EXPECTATION_MINIMUM_MIN_VALUE IS NOT NULL
       AND PROFILE_MIN_NUMERIC < TRY_CAST(CUSTOM_EXPECTATION_MINIMUM_MIN_VALUE AS DECIMAL(38,8)));

PII Detection Summary

SELECT
    DATA_SOURCE_NAME,
    TABLE_NAME,
    COLUMN_NAME,
    CUSTOM_PII_DETECTION_TAGS,
    CUSTOM_SENSITIVITY_LEVEL
FROM REPOSITORY.VW_DATASET_COLUMNS
WHERE CUSTOM_PII_DETECTION_TAGS IS NOT NULL
ORDER BY DATA_SOURCE_NAME, TABLE_NAME, SEQUENCE;

Databricks Notes

Replace TRY_CAST(x AS DECIMAL(38,8)) with TRY_CAST(x AS DECIMAL(38,8)) (same syntax)
Profile percentage columns are already multiplied by 100 in the view (e.g., 50 means 50%, not 0.5)

Documentation Index