Skip to content
The docs are now organized by product area. Old links still work. Report a broken link

Setting Up CI

Get real test results flowing into your published report, using the qualflare-action GitHub Action or the qf CLI directly.

Last updated

A published report with no launches just shows "No runs yet." This page is the fastest path from there to a report that actually reflects your test history — full detail lives in the CLI Tool docs; this is the short version for an open-source project on GitHub Actions.

Some frameworks have a native Qualflare reporter. See Native reporters for the list. A native reporter keeps steps, attachments and retry history that JUnit XML drops, so use one where it exists. For any other framework, have your test runner write a results file it already supports: qf supports 27 frameworks and 4 generic formats. Run qf list-formats to see them all. Either way, an upload step runs after your tests and sends the file or report directory to Qualflare.

A GitHub Action that wraps the CLI — one uses: step instead of a manual binary install:

- name: Run tests
  run: pytest --junitxml=results.xml

- name: Upload results to Qualflare
  if: ${{ !cancelled() }}   # upload even when tests fail
  uses: Qualflare/qualflare-action@v1
  with:
    token: ${{ secrets.QF_TOKEN }}
    project: myapp
    results: results.xml

Put if: ${{ !cancelled() }} on the upload step so it runs when your tests fail. Do not add continue-on-error: true to the test step: it marks the failed step as passed, so the job and the pull request check turn green even when tests fail.

Full reference — every input, the runner support matrix, version pinning — is on the GitHub Actions page.

Option 2: the CLI directly

Works on any CI platform, not just GitHub Actions, and takes every qf <identifier> collect flag, including the ones the Action has no input or variable for, such as --format, --shard and --dry-run:

# qf login reads the token from the QF_TOKEN environment variable
qf login ci --force
qf ci collect results.xml

You don't need the CLI for --environment, --milestone or --branch: with the Action, set QF_ENVIRONMENT, QF_MILESTONE or QF_BRANCH in the job's env:, and qf reads them inside the Action too (What the action doesn't expose).

See CI/CD Integration for the full multi-platform guide (GitHub Actions, GitLab CI/CD, Jenkins, Azure Pipelines, CircleCI, Bitbucket Pipelines and any other CI) and Upload results with qf collect for the full flag reference.

Getting a token

Both options need a Qualflare access token. Mint one at Project Settings → Access Tokens, then store it as a CI secret. Option 1 passes the secret to the Action's token input, so any secret name works. Option 2 needs the job to expose it as the QF_TOKEN environment variable, which qf login reads.

Keep the CI token valid

Every access token expires after 7, 30, 60 or 90 days, as chosen when it is created (30 by default); 90 days is the longest Qualflare allows. Seven days before a token expires, Qualflare emails the person who created it (not for a 7-day token), and Access Tokens marks it Expires soon. After it expires, every upload fails with exit code 3. Note the expiry date where every maintainer can see it, such as a shared calendar, and replace the token before then with Rotate the CI access token.

Pull requests from forks

GitHub passes no secrets to a workflow run for a pull request from a fork, and treats runs that Dependabot starts the same way. The upload step then gets an empty token: qualflare-action fails with Error: token cannot be empty, and the job fails. To skip the upload for those runs, put this if: on the upload step in place of if: ${{ !cancelled() }}, whether it uses the Action or the CLI:

if: ${{ !cancelled() && github.actor != 'dependabot[bot]' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) }}

With it, the upload is skipped only for pull requests from forks, for Dependabot's runs and for cancelled runs. Pushes, pull requests from your repository's own branches, and scheduled and manual runs upload, also when tests fail. Don't switch the workflow to pull_request_target to get the secret: GitHub says to avoid that event if you build or run code from the pull request.

Persistent self-hosted runners

On a self-hosted runner that keeps its home directory between jobs, the saved login stays too. qualflare-action v1.0.3 and later (what @v1 runs) reads the token from QF_TOKEN and logs in with --force, replacing the saved token on every run, and so does qf login ci --force with the CLI directly. A workflow pinned to an earlier Action tag fails from its second run with identifier "myapp" already exists (Identifier already exists). The token stays in qf's credentials file after the job; to keep that file in the job's directory and remove it, see Self-hosted runners.

GitHub says self-hosted runners should almost never be used for public repositories, because anyone can open a pull request against the repository and compromise the runner.

What the report makes public

While the report is published, every launch your CI uploads to the project appears on it, with each test's name, status and failure message (what a public report shows).

  • Search engines can index the report's launches page, and the reports sitemap lists it.
  • Each run's page is noindex, so search engines don't list it, but anyone with its link can open it.
  • The badge and banner are cached for a minute (Caching).
  • When you unpublish, Qualflare stops serving the report at once, but cached pages can stay visible for up to about an hour. Then the launches page answers 404 with noindex, and the sitemap no longer lists it.