Upload test results from any CI system

Upload test results to Qualflare from Travis CI or any other CI with the qf CLI, its Docker image or npx: self-hosted runners and branch detection included.

To upload test results to Qualflare from Travis CI or any other CI system, run qf login <identifier> --force with the token in QF_TOKEN, then qf <identifier> collect <report>, using the qf binary, the ghcr.io/qualflare/qf Docker image or npx @qualflare/cli.

Before you begin

A project access token with Allow writes off, which you create in Project Settings > Access Tokens Editor+ · project (requires the Editor role or higher in the project). See Access Tokens.
A report file your tests write, such as JUnit XML. The examples use ci as the identifier.

Upload from any CI

Store the token as a secret that your CI exposes to the job as the environment variable QF_TOKEN. Run the two commands of How a CI upload works after the tests, in a step that runs even when they fail. collect exits 0 when the upload succeeds, so the test step still decides the job's result. Install qf from a release archive (Binary Download), or run it from the Docker image or npx below.

Branch and commit

qf takes the branch from the first of QF_BRANCH, GIT_BRANCH, GITHUB_REF_NAME, CI_COMMIT_REF_NAME and BITBUCKET_BRANCH that has a value, and the commit from QF_COMMIT, GIT_COMMIT, GITHUB_SHA, CI_COMMIT_SHA and BITBUCKET_COMMIT. On a CI that sets none of them, qf asks the local checkout, and a detached HEAD gives no branch, so set QF_BRANCH and QF_COMMIT from your CI's own variables. The Branch and commit detection table has the rule for each CI.

Travis CI

Store QF_TOKEN in the repository's Settings > Environment Variables, where Travis hides the value in the build log by default. Then add to your .travis.yml:

language: python
python: "3.12"
script:
  - pytest --junitxml=results.xml
after_script:  # runs whether the tests passed or failed
  - VERSION=0.4.0
  - curl -L "https://github.com/qualflare/qualflare-cli/releases/download/v${VERSION}/qf_${VERSION}_linux_amd64.tar.gz" -o qf.tar.gz
  - tar -xzf qf.tar.gz && sudo mv qf /usr/local/bin/
  - export QF_BRANCH="${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}" QF_COMMIT="${TRAVIS_PULL_REQUEST_SHA:-$TRAVIS_COMMIT}"
  - qf login ci --force
  - qf ci collect results.xml

qf reads no TRAVIS_* variable, and Travis checks out the tested commit as a detached HEAD, so the export line sets the branch and commit: the pull request's own on a pull request build, where TRAVIS_BRANCH is the target branch. after_script doesn't change the build result, so check its log for a failed upload.

Docker image

The image has qf but no git, so pass the branch and commit in. It runs as the user qf, so that user must be able to read the mounted results. Each docker run --rm starts with no saved login, so log in and collect in the same container:

docker run --rm \
  -e QF_TOKEN -e QF_BRANCH -e QF_COMMIT -e QF_ENVIRONMENT \
  -v "$(pwd)":/data -w /data --entrypoint sh \
  ghcr.io/qualflare/qf:0.4.0 -c 'qf login ci --force && qf ci collect results.xml'

-e NAME with no value passes the job's own value into the container, and no other variable crosses, so set QF_BRANCH and QF_COMMIT in the job first. More on the image: Docker.

npx

With Node.js 16 or later, npx runs qf from the npm package @qualflare/cli, and downloads the pinned package the first time:

npx --yes @qualflare/cli@0.4.0 login ci --force
npx --yes @qualflare/cli@0.4.0 ci collect results.xml

The package brings the qf binary for Linux, macOS and Windows on x64 and arm64. More: npm.

Self-hosted runners

On a runner that keeps its home directory between jobs, the saved login stays too, so always log in with --force (Identifier already exists). When several jobs share one Linux runner, give each job its own credentials file, so one job's login doesn't replace the token another job saved: set XDG_CONFIG_HOME="$PWD/.qf-config" for both qf commands, and qf keeps the token in .qf-config/qualflare/config.toml under the job's directory (Credentials file).

Pin and check the qf version

The archive, Docker and npx examples pin qf 0.4.0. To pin another release, use a version from the qf releases page. Each release lists the SHA-256 checksum of every archive in checksums.txt, so you can check a download before you install it:

VERSION=0.4.0
curl -LO "https://github.com/qualflare/qualflare-cli/releases/download/v${VERSION}/qf_${VERSION}_linux_amd64.tar.gz"
curl -LO "https://github.com/qualflare/qualflare-cli/releases/download/v${VERSION}/checksums.txt"
sha256sum --check --ignore-missing checksums.txt

--ignore-missing, a GNU sha256sum option, skips the archives you didn't download. checksums.txt is signed with cosign: the signature and certificate are checksums.txt.sig and checksums.txt.pem on the same release page.

Troubleshooting

Error: token cannot be empty means QF_TOKEN didn't reach the step, or, with Docker, the container: add -e QF_TOKEN. Launches with no branch: set QF_BRANCH. identifier "ci" already exists on a reused runner: log in with --force. More fixes: Troubleshooting on CI/CD integration and Troubleshoot qf and CI uploads.

Guides: CI/CD integration · CircleCI · Bitbucket Pipelines
Reference: Install Qualflare CLI · Authenticate with qf login
CLI/API: Access Tokens

Other CI systems, Docker and npx