Skip to main content

Testiny Automation Guide

Testiny works with all test automation tools and frameworks to upload and collect automated test results. With Testiny you can manage, debug and analyze all your tests in a single place. Whether you use a CI/CD pipeline or manually launch your automated tests, collect all automated results in Testiny and efficiently fix CI failures or flaky tests.

Testiny helps you with debugging, troubleshooting and analyzing by providing convenient access to:

  • all your automated test results
  • screenshots, videos or any other attachments and artifacts
  • test result summaries
  • contextual information such as duration, test execution history and environment variables

And the Testiny CLI is designed for different use cases and scenarios:

  • Fast delivery of your test automation results from the command line or your CI/CD pipeline
  • Supports both standard and advanced workflows such as parallel testing jobs
  • Works with any test automation tool and framework by using standard JUnit-style XML report files & other report formats

Concepts​

In this section, we'll explain the concept of sources, automation runs and run fields in Testiny.

Sources​

To easily group your automation test runs, you can specify the source of your test runs when you importing your results with the Testiny CLI.

You would group your test runs by your test suites, for example you might have frontend-unit tests, backend-unit tests, backend-api tests, integration tests, and so on. By grouping your runs by source/test suite, you can properly analyze and compare tests over time.

Make sure, that different test suites are reported to different sources. Different environments, i.e. the same tests but executed with different configurations, are tracked by result fields and not by source.

Test runs & results​

The CLI automatically creates automation test runs in Testiny and they are always assigned to a source. The CLI then imports the automated results, adds them to the test run and completes the test run. No results can be added to completed runs. If you don't want the run to be completed automatically, set the --incomplete flag.

Fields & Run Fields​

The CLI automatically detects your current environment and automatically adds some predefined environment variables as fields to your automated results. For example, if you run the CLI in GitHub Actions, the CLI will add environment variables such as the commit hash, tags, pipeline ID and job ID to the results. You can also add your own fields to your results using the --field-values option. See examples.

The CLI also sets the Run Fields. Run Fields (and of course the specified source) are used to identify test runs and to group together automated results. For example, in CI environments, the Pipeline ID, Build ID, Job ID or any comnbination can be used for Run Fields.

Run Fields are really powerful in CI environments when importing results from split tests (e.g. parallel testing, test sharding, different environments, ...), as you can use the available environment variables to add all the results to the same test run.

To give you a concrete example: In GitLab CI pipelines, the CLI will use the CI_PIPELINE_ID environment variable as the default Run Field and Testiny will then group together all results with the same pipeline ID and the same source. When running the CLI locally, the CLI will by default use an auto-generated number as the Run Field to identify the run and its results. See all examples for CI/CD providers.

For simpler use cases and popular CI environments, you won't need to specify the Run Fields yourself as the CLI detects them automatically. For more complex scenarios such as parallel testing or splitting your tests, or in lesser known CI environents, you may need to specify the Run Fields explicitly.

Test cases​

When uploading results for the first time, Testiny automatically creates the test cases for you. When uploading results from subsequent test runs (from the same test suite and to the same source), Testiny matches the test cases to re-use them. This makes it easy to track the results of a test case over time and to detect flakiness and find bugs.

Test cases are matched first by their key, and then by their title/name. By default, most automation frameworks don't provide a test case ID or key, so the title/name of the test case is used. If the name/title of the automated test case changes (and no key is provided), Testiny would create a new test case.
If you want to manually link your automated test cases to the test cases in Testiny, you would need to add IDs or keys to the test cases in your automation framework.

Retention period​

Testiny automatically deletes old runs for you to avoid clutter and save storage. Runs that are older than the configured retention period will be automatically deleted.

With administrator permissions, you can change the default retention period for all automated runs in Settings > Automation. You can also set different retention periods per source.

Pinning runs​

If you would like to keep certain runs from being deleted, you can pin the automated run. Pinned runs will not be deleted, even if they are older than the defined retention period. If you unpin a run and it is already older than the retention period, it will be queued to be deleted.

CLI Usage Guide​

Submit your first automation results​

  1. Download or install the Testiny CLI and create an API key in Testiny.

  2. Run your automated tests & generate a report. Testiny CLI supports JUnit-style XML files, Playwright JSON report, Cypress JSON reports and more. See the CLI reference for all supported report formats.

  3. Import your automated results to Testiny. Specify the Testiny API key, project ID or key, a source name, the report type and the report file(s):

export TESTINY_API_KEY=****
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --junit report.xml
See command output
API key: user '[email protected]' in organization 'testiny'
Using project: Demo (DE)
Import source: integration-tests
Environment: gitlab
Processing nunit file: report.xml
  Submitted 168/168 results
Completed 1 automation test run(s)
Imported all 1 file(s) successfully
Submitted Testiny automated run ids: 264

The CLI will parse the report file, create an automation run in Testiny and add all results, errors and attachments to the run. The run is added to the specified project. The source name is used to group your test runs together (so choose a name that fits your test suite, e.g. frontend-tests, backend-tests, e2e-tests, api-tests, ...). In the console output, you can see how many test files have been processed and how many results and attachments have been uploaded.

If the CLI is executed in a CI environment, it auto-detects various environment variables like commit hashes and tags, pipeline and job IDs, and adds these fields to your automation run. Also see our examples for CI/CD integration.

Additional Fields​

The importer tries to detect relevant environment variables and fields, and adds these fields automatically to your tests in Testiny. The fields will be shown in Testiny as additional information to your test run and results, and contains details such as the execution environment, the current commit hash or the CI build ID.
If you would like to add additional fields to your test results, use the --field-values option to pass a comma-separated list of key-value pairs:

--field-values my_field=myValue,another_field=anotherValue

You can also use environment variables as values:

--field-values project_title=$CI_PROJECT_TITLE

For known enviroments, the CLI already adds the CI build URL or the CI job URL to the automated test run. If you would like to add more links (e.g. to artifacts, to the repository, ...), you'll simply add it as a field to the run and Testiny will display it as a clickable URL:

--field-values my_url=https://www.example.com

Good to know​

Importing multiple files​

You can import a single file, or multiple files at once. Here's an example to import all XML files in a folder:

export TESTINY_API_KEY=****
npx @testiny/cli automation --project ID_OR_KEY unit-tests --junit results/*.xml

Test Run Title​

The CLI generates a title/name for the test run, which is shown in Testiny. The default title depends on the (CI) environment. You can specify your own title pattern with the --run-title-pattern option. In the pattern, you can use field values (pre-defined fields and your additional fields) to generate unique run titles:

--run-title-pattern "Pipeline %{ci_pipeline_id} - %{vc_branch} - %{vc_commit}"

To see the default run title and automatically detected fields for a specific CI environment, see our examples for CI/CD integration.

Attachments​

The importer parses the reports and detects if there are any files linked to the tests and uploads these files to Testiny. If you would like to disable uploading attachments you can do so by adding the flag --no-attachments.

The importer takes the path to the attachments from the report (if there are any). By default, the base directory for the file paths is the same directory as that of the the report files. If this path does not match the location of the attachments, you can specify the correct path with the option --attachment-path <path/to/attachments>.

If the attachment size exceeds the upload limit of your Testiny account, the CLI will throw the error API_DATA_TOO_LARGE and uploading attachments will be interrupted. The run and results will still be uploaded, and the run will not automatically be marked as completed.

Testiny Server​

If you are using Testiny Server, you'll need to specify the url to your Testiny instance with the --app option.

CI Environment is not detected​

The CLI tries to automatically detect the CI environment its running in. If the auto-detection does not work, you can specify one of the supported environments ("gitlab", "github", "azure", "circle", "jenkins", "teamcity", ...) with the --environment option. Then Testiny uses the predefined environment config to add environment variables as fields to the imported results.

Run Fields​

As explained above, the CLI automatically sets the Run Fields, which are use to identify test runs and to group together automated results. In most CI environments, the CLI uses the Build ID or Pipeline ID as the Run Field. If the CI environment is unknown, it uses an auto-generated fallback ID.

You can override the default Run Fields in more complex scenarios such as parallel testing or splitting your tests, or in unknown CI environents, with the --run-fields option. You can use any field(s) from your automated results as a Run Field and pass them as a comma-separated list of field names:

--field-values pipeline_id=$MY_CI_PIPELINE_ID,my_field=myValue --run-fields pipeline_id,my_field

Parallel or Sharded Test Runs​

If you run your tests in different jobs, for example to run tests with different configurations, or if you split/shard your tests to speed up the execution time, you'll have your test results and reports in different jobs.

Some tools offer an option to merge the reports together in a single report, but you can skip this step when reporting to Testiny. Testiny will group together the results by the Run Fields, so when reporting to Testiny from different jobs you can still group together your results by build ID or pipeline ID. In known CI environments, the Run Fields are automatically set by the CLI. (In unknown CI environments you need to set the Run Fields yourself.)

Reporting results from different jobs, but to the same automated test runs works as follows:

  1. Report results with the --incomplete flag & with the same Run Fields and Run Fields values
  2. Set the run complete after all jobs have finished with the option --complete-runs

As the results are identifed and grouped together by Run Fields (and by source), you don't need to pass around any IDs from Testiny.

Let's go through a concrete example. First, we run the tests in different jobs (in the same build/pipeline) with the --incomplete flag to let Testiny know that more results might be added from another job:

# JOB 1
npx @testiny/cli automation [...] --source unit-backend --incomplete --field-values db_config=pg-legacy

# JOB 2
npx @testiny/cli automation [...] --source unit-backend --incomplete --field-values db_config=pg

Then, we run the automation command with the --complete-runs flag to mark the open runs as complete. The run, that should be completed, is found by matching the current environment's Run Field values (e.g. the pipeline or build ID):

# complete run after jobs have finished:
npx @testiny/cli automation [...] --source unit-backend --complete-runs

Unknown CI Environments​

If the above commands are executed in a known CI environment, the Run Fields are automatically set by the CLI and you don't need to specify them yourself. In an unknown CI environment or more complex scenarios, you need to set the Run Fields yourself (with the option --run-fields) to group results over jobs:

# JOB 1
npx @testiny/cli automation [...] --source unit-backend --incomplete --field-values pipeline_id=$MY_CI_PIPELINE_ID,db_config=pg-legacy --run-fields pipeline_id

# JOB 2
npx @testiny/cli automation [...] --source unit-backend --incomplete --field-values pipeline_id=$MY_CI_PIPELINE_ID,db_config=pg --run-fields pipeline_id
# complete run after jobs have finished:
npx @testiny/cli automation [...] --source unit-backend --complete-runs --field-values pipeline_id=$MY_CI_PIPELINE_ID,db_config=pg --run-fields pipeline_id

Examples & more resources​

Learn how to upload test results from your automation framework and CI/CD provider: