Skip to main content

GitLab CI/CD

Integrate Testiny with GitLab and submit automated test results from your CI/CD deployments to Testiny.

info

Testiny also integrates with GitLab for defect & requirement management.

You can use any test automation framework with Testiny, you'll just have to configure your tests to generate a result file in one of the supported file formats (JUnit XML, TestNG, Playwright JSON, NUnit and more). The test results are then submitted with the Testiny CLI (available as npm package or standalone binary). With Testiny, you can track your tests over time, identify frequently failing tests, efficiently debug and fix CI failures or flaky tests, and have acccess to all test results in a single place.

General Workflow​

  1. Install the Testiny CLI. Typically, you would install the CLI with NPM, making it easy to install and to keep it up-to-date.
  2. Execute your automated tests within your GitLab pipeline and generate a test result report in one of the supported formats.
  3. Use the Testiny CLI to upload the results to Testiny.
info

You don't need to use Node.js for your project or automated tests, but you can conveniently install the Testiny CLI as a npm package via Node.js. This way you can easily upgrade the CLI or always use the latest version. If you do not want to use Node.js, you can also use the Testiny CLI standalone binaries to upload test results.

Basic Example​

This example shows how to upload your automated test results to Testiny from your GitLab pipeline. In this example, we show how to upload test results from Playwright, but you can use any other automation framework.

In this example, we install the Testiny CLI via npm, run the automated test and finally, upload the results with the CLI. The CLI automatically adds some environment variables from the GitLab CI environment to the test run. See the list below for all used environment variables.

You'll also need an API key to authenticate with Testiny. Learn how to create an API key in Testiny.

caution

Please note, that the Testiny API key is not stored in the code, but as a CI/CD variable in GitLab. Define a variable TESTINY_API_KEY and it will be automatically used by the CLI.

.gitlab-ci.yml
stages:
  - tests

run-tests:
  stage: tests
  image: node:latest
  script:
    # Install project dependencies
    # ...
    # Run your automated tests & generate a report file
    # ...
  after_script:
    # Install Testiny CLI & upload results
    # Note, that the API key is stored in the GitLab CI/CD variable TESTINY_API_KEY.
    - |
        npm install --no-save @testiny/cli@latest
        npx @testiny/cli automation --project 1 --source "e2e-tests" --playwright results/results.json

For projects with multiple test suites — such as frontend, backend, or e2e-tests — you can report each suite as a separate test run within the same GitHub Actions pipeline. Simply execute the CLI command npx -y @testiny/cli automation (or testiny-importer automation) for each test suite. Make sure you use different source names (otherwise, Testiny would group together the results from the different test suites).

More CLI Options​

The CLI also offers option to add additional fields with --field-values or to specify a custom run title with --run-title-pattern.
If you're using Testiny Server, add the URL to your instance to the CLI command with the --app option.

Learn more about the CLI in our CLI Usage Guide.

Environment Variables​

Following environment variables are automatically collected by the Testiny CLI for GitLab:

ci_pipeline_id = CI_PIPELINE_ID
ci_job_id = CI_JOB_ID
ci_job_name = CI_JOB_NAME
ci_project = CI_PROJECT_PATH_SLUG
ci_runner_id = CI_RUNNER_ID
ci_runner_desc = CI_RUNNER_DESCRIPTION
ci_build_url = CI_JOB_URL
vc_branch = CI_COMMIT_BRANCH or CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
vc_commit = CI_COMMIT_SHORT_SHA
vc_tag = CI_COMMIT_TAG

Default Run Fields = [ci_pipeline_id]

Default runTitlePattern = "Pipeline %{ci_pipeline_id} - %{vc_branch} - %{vc_commit}"

Parallel/Sharded Test Runs Example​

Testiny also supports reporting results from parallel or sharded test runs. Testiny groups together automated results by Run Fields and by Source. In GitLab, results with the same CI_PIPELINE_ID, and from the same source are automatically grouped together. Therefore, when splitting your tests over multiple jobs, you just need to import the results with the same source name and mark them as incomplete (as more results are going to be added to the run). After all results have been submitted, mark the test run in Testiny as completed.

The following example shows how to report the results from sharded Playwright tests to the same automation run in Testiny:

.gitlab-ci.yml
stages:
  - tests

run-tests:
  stage: tests
  image: node:latest
  parallel: 2
  script:
    # Install project dependencies
    # ...
    # Run your automated tests in parallel & generate a report file
    # ...
  after_script:
    # Install Testiny CLI & upload results
    # Note, that the API key is stored in the GitLab CI/CD variable TESTINY_API_KEY.
    - |
        npm install --no-save @testiny/cli@latest
        npx @testiny/cli automation --project 1 --source "e2e-tests" --playwright results/results.json --incomplete
      
complete-tests:
  stage: tests
  image: node:latest
  needs: 
    - run-tests
  script:
    - |
        npm install --no-save @testiny/cli@latest
        npx @testiny/cli automation --project 1 --source "e2e-tests" --complete-runs

More resources​