Azure Pipelines
Integrate Testiny with Azure Pipelines and submit automated test results from your CI/CD deployments to Testiny.
Testiny also integrates with Azure DevOps for defect & requirement management.
You can use any test automation framework with Testiny, you'll just have to configure your tests to generate a result file in one of the supported file formats (JUnit XML, TestNG, Playwright JSON, NUnit and more). The test results are then submitted with the Testiny CLI (available as npm package or standalone binary). With Testiny, you can track your tests over time, identify frequently failing tests, efficiently debug and fix CI failures or flaky tests, and have acccess to all test results in a single place.
General Workflow
- Install the Testiny CLI. Typically, you would install the CLI with NPM, making it easy to install and to keep it up-to-date.
- Execute your automated tests within your Azure Pipelines and generate a test result report in one of the supported formats.
- Use the Testiny CLI to upload the results to Testiny.
You don't need to use Node.js for your project or automated tests, but you can conveniently install the Testiny CLI as a npm package via Node.js. This way you can easily upgrade the CLI or always use the latest version. If you do not want to use Node.js, you can also use the Testiny CLI standalone binaries to upload test results.
Setup Example
This example shows how to upload your automated test results to Testiny from Azure Pipelines. In this example, we show how to upload test results from Playwright, but you can use any other automation framework.
In this example, we install the Testiny CLI via npm, run the automated test and finally, upload the results with the CLI. The CLI automatically adds some environment variables from the CI environment to the test run. See the list below for all used environment variables.
You'll also need an API key to authenticate with Testiny. Learn how to create an API key in Testiny.
Please note, that the Testiny API key is not stored in the code, but as a secret variable in Azure Pipelines.
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- task: NodeTool@0
inputs:
versionSpec: '18'
displayName: 'Install Node.js'
- script:
# Install project dependencies
# ...
# Run your automated tests & generate a report file
# ...
# Install Testiny CLI & upload results
- task: PublishTestResults@1
condition: succeededOrFailed()
- script: npm install --no-save @testiny/cli@latest
displayName: 'Install Testiny CLI'
- script: npx @testiny/cli automation --project 1 --source e2e-tests --playwright results/results.json
displayName: 'Upload results to Testiny'
env:
TESTINY_API_KEY: $(SECRET_TESTINY_API_KEY)
For projects with multiple test suites — such as frontend, backend, or e2e-tests — you can report each suite as a separate test run within the same GitHub Actions pipeline. Simply execute the CLI command npx -y @testiny/cli automation (or testiny-importer automation) for each test suite. Make sure you use different source names (otherwise, Testiny would group together the results from the different test suites).
More CLI Options
The CLI also offers option to add additional fields with --field-values or to specify a custom run title with --run-title-pattern.
If you're using Testiny Server, add the URL to your instance to the CLI command with the --app option.
Learn more about the CLI in our CLI Usage Guide.
Environment Variables
Following environment variables are automatically collected by the Testiny CLI for GitHub:
ci_build = BUILD_BUILDID
ci_project = SYSTEM_TEAMPROJECT
ci_repository = BUILD_REPOSITORY_NAME
ci_agent_id = AGENT_ID
ci_pipeline = BUILD_DEFINITIONNAME
ci_build_url = `${SYSTEM_TEAMFOUNDATIONCOLLECTIONURI}${SYSTEM_TEAMPROJECT}/_build/results?buildId=${BUILD_BUILDID}&_a=summary`
vc_ref = BUILD_SOURCEBRANCHNAME
vc_commit = BUILD_SOURCEVERSION
Default Run Fields = [ci_build]
Default runTitlePattern = "%{ci_pipeline} - %{ci_build} - %{vc_ref} - %{vc_commit}"
Parallel/Sharded Test Runs Example
Testiny also supports reporting results from parallel or sharded test runs. Testiny groups together automated results by Run Fields and by Source. In Azure Pipelines, results with the same BUILD_BUILDID, and from the same source are automatically grouped together. Therefore, when splitting your tests over multiple jobs, you just need to import the results with the same source name and mark them as incomplete (as more results are going to be added to the run). After all results have been submitted, mark the test run in Testiny as completed.
The following example shows how to report the results from sharded Playwright tests to the same automation run in Testiny:
trigger:
- main
pool:
vmImage: ubuntu-latest
jobs:
- job: run-tests
strategy:
matrix:
chromium-1:
project: chromium
shard: 1/2
chromium-2:
project: chromium
shard: 2/2
firefox-1:
project: firefox
shard: 1/2
firefox-2:
project: firefox
shard: 2/2
steps:
- task: NodeTool@0
inputs:
versionSpec: '18'
displayName: 'Install Node.js'
- script:
# Install project dependencies
# ...
# Run your automated tests & generate a report file
# ...
# Example: npx playwright test --project=$(project) --shard=$(shard)
- task: PublishTestResults@1
condition: succeededOrFailed()
# Install Testiny CLI & upload results
- script: npm install --no-save @testiny/cli@latest
displayName: 'Install Testiny CLI'
- script: npx @testiny/cli automation --project 1 --source "e2e-tests" --playwright results/results.json --incomplete
displayName: 'Upload results to Testiny'
env:
TESTINY_API_KEY: $(SECRET_TESTINY_API_KEY)
- job: complete-tests
dependsOn: run-tests
condition: succeededOrFailed()
steps:
# Complete test run
- task: CompleteTestRun@1
condition: succeededOrFailed()
- script: npm install --no-save @testiny/cli@latest
displayName: 'Install Testiny CLI'
- script: npx @testiny/cli automation --project 1 --source "e2e-tests" --complete-runs
displayName: 'Complete tests'
env:
TESTINY_API_KEY: $(SECRET_TESTINY_API_KEY)