Overview
The Jira Cloud (API Token) data source template connects Validatar to a Jira Cloud site. A data source built from it does three jobs:
- Opens and resolves issues from test results, through test actions and a reconciliation sweep.
- Catalogs the Jira site: projects, issue types with the fields they require, site-wide reference lists, and the issues Validatar raised.
- Makes Jira data testable, through read macros for JQL, issue counts, aging, time in status, field fill rates and change history.
This page documents the template's parameters, its catalog and its 40 macros. For what the integration does end to end, start with Jira Integration. For setup, see Jira Setup for Validatar.
Marketplace: Jira Cloud - Data Source Template
How it works
The template is a Python script data source. Its pre-execution script holds the Jira client, the Validatar API client and the issue lifecycle, and Validatar prepends it to every macro, test script and ingestion run on the data source. It uses requests and pandas only, so nothing has to be installed where Validatar runs Python.
Calls go to Jira's REST API version 3. Searches use /rest/api/3/search/jql, which replaced the search endpoint Atlassian has removed. Rate limits (HTTP 429) and server errors are retried with the delay Jira asks for. Any other error from Jira stops the run with Jira's own message, which usually names the field or permission at fault.
Authentication
| Auth Mode | Use with | Base address |
|---|---|---|
basic (default) |
An account email and a classic API token | Your site URL |
basic_scoped |
An account email and a scoped API token | https://api.atlassian.com/ex/jira/<cloudId>, found automatically from the site URL |
pat |
A Jira Data Center personal access token | Your server URL. Not yet tested; Jira Data Center is not supported yet |
An API token carries every permission of the account it belongs to. Limit what the integration can do by limiting the account, as Jira Setup for Validatar describes.
Connection parameters
Jira
| Parameter | Default | Description |
|---|---|---|
| Jira Site URL | Required. Your Jira Cloud site, such as https://your-domain.atlassian.net, with no trailing slash |
|
| Auth Mode | basic |
Required. See Authentication |
| Account Email | The Atlassian account the token belongs to. Required for basic and basic_scoped |
|
| API Token | Required. Stored as a secret | |
| API Version | 3 |
3 sends rich text as Atlassian Document Format. 2 sends plain text |
| Project Keys | Comma-separated keys of the projects to catalog and allow, such as DQ,FIN. Blank catalogs every project the account can see, which on a large site is slow |
|
| Default Project Key | Where new issues go unless a test's Issue Project field or the action says otherwise | |
| Default Issue Type | Bug |
The issue type new issues are created as |
| Field Defaults (JSON) | Values for fields every new issue needs, by field name or id, such as {"Data Source": "Snowflake PROD", "DQ Severity": "High"} |
|
| Resolve Status | A status to resolve into. Blank takes the first transition into a done status, whatever the workflow calls it | |
| Request Timeout (seconds) | 60 |
Per request |
| Max Retries | 4 |
Retries on HTTP 429 and 5xx |
Validatar
| Parameter | Default | Description |
|---|---|---|
| Validatar Base URL | This Validatar instance, such as https://acme.cloud.validatar.com. Needed for the Automatic action macros, the Issue Tracking fields on tests, the sweep and the health checks |
|
| Validatar API Token | A token with the Authoring, Data Steward and Explorer scopes. Stored as a secret | |
| Validatar API Path Prefix | /core/api/v1 |
Change only if your instance serves its API elsewhere |
| Validatar Instance Key | A short name for this Validatar instance in Jira labels. Blank uses the first part of the host name, such as acme. Set it when two Validatar instances share one Jira site under similar host names |
Lifecycle
| Parameter | Default | Description |
|---|---|---|
| Dry Run | on | Actions and the sweep decide and write nothing, to Jira or to Validatar. Turn off when the setup is proven |
| Issue Grain | incident |
incident: one open issue per test. failure: a new issue for every failing run |
| Open After Failures | 1 |
Open an issue only after this many failures in a row |
| Cooldown Hours | 0 |
A failure within this many hours of a close comments on the closed issue instead of opening a new one. 0 switches it off |
| Resolve After Passes | 1 |
Resolve after this many passes in a row. 0 leaves the close to a person and notes the pass on the issue once |
| Error Policy | failure |
failure handles an Error result like Failed. ignore takes no action on errors |
| Escalate After | 0 |
Raise the issue's priority one step after this many occurrences. 0 switches it off |
| Reconcile Data Source Ids | Comma-separated ids of the warehouse data sources whose linked tests the sweep and the health checks cover | |
| Reconcile Settle Seconds | 30 |
The sweep waits this long, once, before opening or resolving anything, so actions from the same job land first |
An action can override Issue Grain and Resolve Status for one test through its own parameters.
Catalog
| Parameter | Default | Description |
|---|---|---|
| Ingest Issues | on | Catalog the issues Validatar raised, under a <KEY> Issues schema per project |
| Ingest Issue Days | 30 |
Closed issues stay in the catalog this many days after closing |
| Ingest Issue Limit | 1000 |
Most issues cataloged across all projects |
| Ingest Issues JQL | Extra JQL that narrows the issues cataloged, such as priority in (Highest, High) |
Metadata ingestion
Ingest Technical Metadata catalogs the Jira site. The values land in the Jira catalog custom fields; import Jira - Catalog Custom Fields first.
| Schema | Tables | Columns |
|---|---|---|
| One per project, named by its key | One per issue type that the account can create, with its id, description, field count, required field count, and the required fields that have no default | One per field on the create screen: its id, whether it is required, whether Jira fills a default, its type, its allowed values and its update operations |
_config |
fields, statuses, status_categories, priorities, resolutions, each with its row count |
The attributes of each list |
<KEY> Issues, one per project with Validatar issues |
One per issue Validatar raised, named by its key: summary, status, status category, priority, assignee, created, resolved, age in days, occurrences, the Validatar test and the correlation key | The same values as columns, with each value in the column's comment |
The table and column names follow Jira, so a test can name DQ.Bug and its DQ Severity column directly.
Note: Jira's field lookup needs the Create Issues permission. A project the account can browse but not create issues in appears in the catalog with no issue types, and its Jira Ingestion Note field says why.
Ingestion makes one call per project for its issue types, one per issue type for its fields, and a few for the site-wide lists. Set Project Keys on large sites.
Macros
Test actions
Use these in a Run a Macro action step. Each comes in two versions:
- Automatic reads the finished run from Validatar, so the step needs no template fields. Its only parameters are optional overrides: Assignee Email on the macros that open or update an issue, and the project and issue type. It needs Validatar Base URL and Validatar API Token, and the token's user needs access to the test's project.
- Custom takes the run's values as parameters, which you fill with template fields such as
{{result}}. It also takes per-action options the Automatic version leaves to the test's fields and the data source: the summary and description text, issue grain, signature, extra labels and resolve status.
Open and Resolve Jira Issues from Test Results shows both.
| Macro | Reference keys, Automatic and Custom | Does |
|---|---|---|
| Sync Jira Issue with Test Result | sync_issue_auto, sync_issue |
Opens, comments on or resolves the test's issue from its result. The one most tests need |
| Open a Jira Issue | open_issue_auto, open_issue |
Opens an issue when the test has none. Never comments or resolves |
| Comment on the Linked Jira Issue | comment_on_issue_auto, comment_on_issue |
Comments on the open issue. Never opens one |
| Resolve the Linked Jira Issue | resolve_issue_auto, resolve_issue |
Resolves the open issue |
| Update the Linked Jira Issue | update_issue_auto, update_issue |
Sets field values on the open issue, with no comment |
| Link an Existing Jira Issue to this Test | link_issue_to_test_auto, link_issue_to_test |
Adopts an issue a person raised: adds the labels, state and link back, so the lifecycle manages it from then on |
Both versions build the same issue: the test, project, folder, result, counts, completion time, the test's description, a link to the run, any errors, and the scripts that ran on the test and control sides. The Automatic version writes the completion time in UTC; the Custom version writes it in the time zone of the user who ran the test.
Each macro has a Plan twin for a test data set (plan_sync_issue_auto, plan_sync_issue and so on). It runs the same code with writing switched off and returns one row: the decision, the reason, and the full payload Jira would receive. A Plan (Automatic) macro takes the address of a test result page in Execution URL. Plan macros never write, whatever Dry Run says.
Assignee Email takes the email address of a Jira user. Sync and Open assign each new issue to that user; left blank, the test's Issue Assignee field applies, and with neither the issue is created unassigned. Update assigns the open issue to that user, and leaves the assignee as it is when blank. Comment, Resolve and Link never open an issue, so they have no assignee.
On an action step, Preview of an Automatic macro reads and writes nothing, because a preview has no run behind it. Preview of a Custom macro runs it for real with a sample result of Passed.
Reconciliation
| Macro | Reference key | Does |
|---|---|---|
| Reconcile Jira Issues with Validatar | reconcile_issues |
For a test data set placed last in a job. Opens issues actions missed, resolves issues whose test passes, closes duplicates, refreshes the Issue Key on tests, and writes the Issue Tracking fields onto linked tables and columns. Returns one row per thing it checked and what it did |
Its parameters narrow the sweep to some data sources, one project or one folder, and turn the catalog writes off.
Reading Jira
For test data sets. Each returns a dataframe.
| Macro | Reference key | Returns |
|---|---|---|
| Query (JQL) | query |
The issues a JQL query matches, with the fields you name |
| JQL Query | jql |
The same, kept for existing tests |
| Issue Count | issue_count |
How many issues a JQL query matches |
| Issue Data | issue_data |
The issues of one project and issue type, both picked from the catalog, with an optional extra JQL filter |
| Open Validatar-Raised Issues by Test | open_issues_by_test |
Open issues Validatar raised, one row per issue with the id of its test |
| Open Issue Aging | issue_aging |
Open issues in a project with their priority, assignee, days open and an age bucket, optionally only those older than a number of days |
| Time in Status | time_in_status |
Hours each issue a query matches spent in each status, from its change history |
| Field Fill Rate | field_fill_rate |
For one issue type, how often each field is filled across a sample of issues |
| Issue Change History | changelog |
One issue's changes, optionally for one field |
| Transitions Available on an Issue | transitions_available |
Where an issue can move next, the status category of each target, and whether the transition opens a screen |
| Required Fields for an Issue Type | required_fields_for_issue_type |
The fields Jira requires to create an issue, live from Jira |
| Plan: Preview a Jira Issue Payload | preview_issue_payload |
The payload a create would send for given field values, checked against the create screen. Writes nothing |
| List of Projects | list_of_projects |
Projects visible to the account |
| List of Issue Types in a Project | list_of_issue_types |
Issue types the account can create in a project |
| Validatar Test Custom Fields | validatar_test_custom_fields |
A Validatar test's custom field values, read through the Validatar API |
Health checks
The pre-execution script also holds the functions behind the Jira Health Checks pack. They are called from the pack's test scripts rather than through macros.
Installation
- Import Jira Cloud - Data Source Template from Marketplace in Validatar. It appears under Settings > Data Source Templates.
- Import the Issue Tracking and Jira custom field sets the same way.
- Create a data source from the template, fill in the parameters, and add it to the projects whose tests will open issues.
- Run Ingest Technical Metadata and check the project's issue type tables.
Jira Setup for Validatar gives each step in full, including the Jira side.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
No creatable issue types returned for project 'DQ' |
The account lacks Create Issues in DQ. Jira answers with an empty list, not an error. Grant the permission |
Cannot create a Bug in DQ: required field(s) have no value |
Add the listed fields to Field Defaults (JSON). The message gives each field's type and allowed values |
These field(s) are not available on DQ / Bug |
A name in Field Defaults, a test's Issue Field Values or the action's Field Values is not on the create screen. Check the spelling against the issue type's catalog table |
Field name '...' is ambiguous |
Two Jira fields share the name. Use the field id, such as customfield_10062 |
Could not read a project id and test id from test_url |
A Custom action's Test URL parameter is not exactly {{test_url}} |
The (Automatic) macros read the run from Validatar |
Validatar Base URL or Validatar API Token is blank on the data source. Set both, or use the Custom macro |
Validatar answered HTTP 403 to Execution/TestExecutionView/... |
The token's user has no access to the test's project. Give that user a role in the project |
WARNING: run lookup: scripts not read |
The issue was opened without the scripts, because Validatar did not return them. The rest of the issue is complete |
No Jira user found for 'pat@example.com' among the users assignable in DQ |
The address is wrong, or the person lacks the Assignable User permission in DQ. The integration looks assignees up among the project's assignable users, which also confirms Jira will accept them. A scoped token needs read:jira-user |
'pat@example.com' matches more than one Jira user |
Enter that user's Jira account id instead of the email |
| Jira refuses the assignee when the issue is created | The integration's account lacks Assign Issues, or the person lacks Assignable User in that project. Grant the permission, or leave Assignee Email blank |
| An issue is opened for every run of a failing test | Issue Grain is failure, or the test's labels were removed in Jira. The Jira Health Checks find both |
| A preview on the action step resolved an issue | The step uses a Custom macro, and Preview runs it for real with a result of Passed. Switch to the Automatic macro, whose Preview writes nothing, or preview only while Dry Run is on |