Skip to main content

GitHub Actions

Integrate Testiny with GitHub and submit automated test results from your CI/CD deployments to Testiny.

info

Testiny also integrates with GitHub for defect & requirement management.

You can use any test automation framework with Testiny, you'll just have to configure your tests to generate a result file in one of the supported file formats (JUnit XML, TestNG, Playwright JSON, NUnit and more). The test results are then submitted with the Testiny CLI (available as npm package or standalone binary). With Testiny, you can track your tests over time, identify frequently failing tests, efficiently debug and fix CI failures or flaky tests, and have acccess to all test results in a single place.

General Workflow​

  1. Install the Testiny CLI. Typically, you would install the CLI with NPM, making it easy to install and to keep it up-to-date.
  2. Execute your automated tests within your GitHub Actions workflow and generate a test result report in one of the supported formats.
  3. Use the Testiny CLI to upload the results to Testiny.
info

You don't need to use Node.js for your project or automated tests, but you can conveniently install the Testiny CLI as a npm package via Node.js. This way you can easily upgrade the CLI or always use the latest version. If you do not want to use Node.js, you can also use the Testiny CLI standalone binaries to upload test results.

Setup Example​

This example shows how to upload your automated test results to Testiny from GitHub Actions. In this example, we show how to upload test results from Playwright, but you can use any other automation framework.

In this example, we install the Testiny CLI via npm, run the automated test and finally, upload the results with the CLI. The CLI automatically adds some environment variables from the GitHub CI environment to the test run. See the list below for all used environment variables.

You'll also need an API key to authenticate with Testiny. Learn how to create an API key in Testiny.

caution

Please note, that the Testiny API key is not stored in the code, but as a secret variable in GitHub Actions.

.github/workflows/tests.yml
name: Automated Tests
on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]
jobs:
  run-tests:
    timeout-minutes: 10
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: lts/*
        cache: npm
    
    - name: Install dependencies
      # Install your dependencies for your project:
      # ...
      # Install Testiny CLI
      run: npm install --no-save @testiny/cli@latest

    - name: Run e2e tests
      # Run your automated tests & generate a report file
      # ...
      
    - name: Upload results to Testiny
      if: ${{ !cancelled() }}

      env: 
        TESTINY_API_KEY: ${{ secrets.TESTINY_API_KEY }}

      # Run CLI to upload results
      run: npx @testiny/cli automation --project 1 --source e2e-tests --playwright results/results.json

For projects with multiple test suites — such as frontend, backend, or e2e-tests — you can report each suite as a separate test run within the same GitHub Actions pipeline. Simply execute the CLI command npx -y @testiny/cli automation (or testiny-importer automation) for each test suite. Make sure you use different source names (otherwise, Testiny would group together the results from the different test suites).

More CLI Options​

The CLI also offers option to add additional fields with --field-values or to specify a custom run title with --run-title-pattern.
If you're using Testiny Server, add the URL to your instance to the CLI command with the --app option.

Learn more about the CLI in our CLI Usage Guide.

Environment Variables​

Following environment variables are automatically collected by the Testiny CLI for GitHub:

ci_run_id = GITHUB_RUN_ID
ci_job = GITHUB_JOB
ci_retry = GITHUB_RUN_ATTEMPT
ci_repository = GITHUB_REPOSITORY
ci_runner = RUNNER_NAME
ci_build_url = `${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}`
ci_workflow_name = GITHUB_WORKFLOW
vc_ref = GITHUB_REF_NAME
vc_commit = GITHUB_SHA

Default Run Fields = [ci_repository, ci_run_id]

Default runTitlePattern = "%{ci_workflow_name} - %{vc_ref} - %{vc_commit}"

Parallel/Sharded Test Runs Example​

Testiny also supports reporting results from parallel or sharded test runs. Testiny groups together automated results by Run Fields and by Source. In GitHub Actions, results with the same GITHUB_REPOSITORY and GITHUB_RUN_ID, and from the same source are automatically grouped together. Therefore, when splitting your tests over multiple jobs, you just need to import the results with the same source name and mark them as incomplete (as more results are going to be added to the run). After all results have been submitted, mark the test run in Testiny as completed.

The following example shows how to report the results from sharded Playwright tests to the same automation run in Testiny:

.github/workflows/tests.yml
name: Automated Tests
on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]
jobs:
  run-tests:
    timeout-minutes: 10
    runs-on: ubuntu-latest
    # runs tests in parallel:
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]

    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: lts/*
        cache: npm
    
    - name: Install dependencies
      # Install your dependencies for your project
      # ...
      # Install Testiny CLI
      run: npm install --no-save @testiny/cli@latest
    
    - name: Run e2e tests
      # Run your automated tests & generate a report file
      # ...
      # Example: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

    - name: Upload results to Testiny
      if: ${{ !cancelled() }}

      env: 
        TESTINY_API_KEY: ${{ secrets.TESTINY_API_KEY }}
      
      # Run CLI to upload results (mark test run as incomplete; results will be reported to the same run)
      run: npx @testiny/cli automation --project 1 --source e2e-tests --playwright results/results.json --incomplete

  complete-tests:
    needs: [run-tests]
    if: always()
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: lts/*
        cache: npm
        
    - name: Install dependencies
      # Install Testiny CLI
      run: npm install --no-save @testiny/cli@latest
    
    # Mark test run as complete
    - name: Complete test run
      env: 
        TESTINY_API_KEY: ${{ secrets.TESTINY_API_KEY }}
      
      run: npx @testiny/cli automation --project 1 --source e2e-tests --complete-runs

More resources​