Setting Up CI
Get real test results flowing into your published report, using the qualflare-action GitHub Action or the qf CLI directly.
Last updated
A published report with no launches just shows "No runs yet." This page is the fastest path from there to a report that actually reflects your test history — full detail lives in the CLI Tool docs; this is the short version for an open-source project on GitHub Actions.
Some frameworks have a native Qualflare reporter. See
Native reporters for the list. A native
reporter keeps steps, attachments and retry history that JUnit XML drops, so
use one where it exists. For any other framework, have your test runner write a
results file it already supports: qf supports 27 frameworks and 4 generic formats. Run
qf list-formats to see them all. Either way, an upload step runs after your
tests and sends the file or report directory to Qualflare.
Option 1: qualflare-action (recommended for GitHub Actions)
A GitHub Action that wraps the CLI — one uses: step instead of a manual
binary install:
- name: Run tests
run: pytest --junitxml=results.xml
- name: Upload results to Qualflare
if: ${{ !cancelled() }} # upload even when tests fail
uses: Qualflare/qualflare-action@v1
with:
token: ${{ secrets.QF_TOKEN }}
project: myapp
results: results.xmlPut if: ${{ !cancelled() }} on the upload step so it runs when your tests
fail. Do not add continue-on-error: true to the test step: it marks the failed
step as passed, so the job and the pull request check turn green even when
tests fail.
Full reference — every input, the runner support matrix, version pinning — is on the GitHub Actions page.
Option 2: the CLI directly
Works on any CI platform, not just GitHub Actions, and takes every
qf <identifier> collect flag, including the ones the Action has no input
or variable for, such as --format, --shard and --dry-run:
# qf login reads the token from the QF_TOKEN environment variable
qf login ci --force
qf ci collect results.xmlYou don't need the CLI for --environment, --milestone or --branch:
with the Action, set QF_ENVIRONMENT, QF_MILESTONE or QF_BRANCH in the
job's env:, and qf reads them inside the Action too
(What the action doesn't expose).
See CI/CD Integration for the full multi-platform guide (GitHub Actions, GitLab CI/CD, Jenkins, Azure Pipelines, CircleCI, Bitbucket Pipelines and any other CI) and Upload results with qf collect for the full flag reference.
Getting a token
Both options need a Qualflare access token. Mint one at Project Settings →
Access Tokens, then store it as a CI secret. Option 1 passes the secret to
the Action's token input, so any secret name works. Option 2 needs the job to
expose it as the QF_TOKEN environment variable, which qf login reads.
Keep the CI token valid
Every access token expires after 7, 30, 60 or 90 days, as chosen when it is created (30 by default); 90 days is the longest Qualflare allows. Seven days before a token expires, Qualflare emails the person who created it (not for a 7-day token), and Access Tokens marks it Expires soon. After it expires, every upload fails with exit code 3. Note the expiry date where every maintainer can see it, such as a shared calendar, and replace the token before then with Rotate the CI access token.
Pull requests from forks
GitHub passes no secrets to a workflow run for a pull request from a fork,
and treats runs that Dependabot starts the same way. The upload step then
gets an empty token: qualflare-action fails with
Error: token cannot be empty, and the job fails. To skip the upload for
those runs, put this if: on the upload step in place of
if: ${{ !cancelled() }}, whether it uses the Action or the CLI:
if: ${{ !cancelled() && github.actor != 'dependabot[bot]' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) }}With it, the upload is skipped only for pull requests from forks, for
Dependabot's runs and for cancelled runs. Pushes, pull requests from your
repository's own branches, and scheduled and manual runs upload, also when
tests fail.
Don't switch the workflow to pull_request_target to get the secret:
GitHub says to avoid that event if you build or run code from the pull
request.
Persistent self-hosted runners
On a self-hosted runner that keeps its home directory between jobs, the
saved login stays too. qualflare-action v1.0.3 and later (what @v1 runs)
reads the token from QF_TOKEN and logs in with --force, replacing the
saved token on every run, and so does qf login ci --force with
the CLI directly. A workflow pinned to an
earlier Action tag fails from its second run with
identifier "myapp" already exists
(Identifier already exists).
The token stays in qf's credentials file after the job; to keep that file in the job's directory and remove it, see
Self-hosted runners.
GitHub says self-hosted runners should almost never be used for public repositories, because anyone can open a pull request against the repository and compromise the runner.
What the report makes public
While the report is published, every launch your CI uploads to the project appears on it, with each test's name, status and failure message (what a public report shows).
- Search engines can index the report's launches page, and the reports sitemap lists it.
- Each run's page is
noindex, so search engines don't list it, but anyone with its link can open it. - The badge and banner are cached for a minute (Caching).
- When you unpublish, Qualflare stops serving the report at once, but
cached pages can stay visible for up to about an hour. Then the launches
page answers
404withnoindex, and the sitemap no longer lists it.
Related
- GitHub Actions — qualflare-action's inputs, outputs and caveats
- CI/CD Integration — the CLI-based guide for every CI platform
- Publishing a Report — where these results end up
Publishing a Report
Turn on a project's public report, embed a badge or banner in your README, and understand what happens if your repository ever loses eligibility.
Open Source FAQ
Common questions about the Open Source plan — rejected repositories, reports going dark, moving projects, and badge vs. banner.