CircleCI
Upload test results from CircleCI to Qualflare with the qf CLI: a config.yml step that runs when tests fail, a QF_TOKEN variable, and QF_BRANCH from CircleCI.
Last updated
To upload test results from CircleCI to Qualflare, add a step with when: always that runs qf <identifier> collect <report>, with the access token in a project environment variable named QF_TOKEN; qf reads no CircleCI variables, so set QF_BRANCH and QF_COMMIT yourself.
Before you begin
- A project access token with Allow writes off, which you create in Project Settings > Access Tokens Editor+ · project (requires the Editor role or higher in the project). See Access Tokens.
- A report file your test step writes, such as JUnit XML. The examples use
cias the identifier.
Store the token as an environment variable
- In CircleCI, open your project's Project Settings > Environment Variables
- Click Add Variable
- Name:
QF_TOKEN - Value: Your Qualflare access token
- Save the variable
A context that holds QF_TOKEN, attached to the job with context: in the workflow, works the same way; with parallel jobs, attach it to the upload job too.
Add the upload step
Add to your .circleci/config.yml:
version: 2.1
jobs:
test:
docker:
- image: cimg/python:3.12
environment:
QF_ENVIRONMENT: staging # must be an environment that exists in the project
steps:
- checkout
- run:
name: Run tests
command: pytest --junitxml=results.xml
- run:
name: Upload results to Qualflare
when: always # upload even when the tests fail
command: |
VERSION=0.4.0
curl -L "https://github.com/qualflare/qualflare-cli/releases/download/v${VERSION}/qf_${VERSION}_linux_amd64.tar.gz" -o qf.tar.gz
tar -xzf qf.tar.gz && sudo mv qf /usr/local/bin/
export QF_BRANCH="$CIRCLE_BRANCH" QF_COMMIT="$CIRCLE_SHA1"
qf login ci --force
qf ci collect results.xml
workflows:
test:
jobs:
- testThe image needs curl, tar and sudo, which the cimg images have.
Branch and commit
qf reads no CircleCI variable, so the export line in the upload step sets QF_BRANCH and QF_COMMIT from CIRCLE_BRANCH and CIRCLE_SHA1. It runs in the shell because CircleCI doesn't expand variables under environment:. On a tag build CIRCLE_BRANCH is empty: qf skips an empty variable, and the local checkout decides. The Branch and commit detection table lists what qf reads on each CI.
Run the upload when tests fail
when: always on the upload step runs it after a failed test step. qf <identifier> collect exits 0 when the upload succeeds, so the failed test step still fails the job.
Parallel jobs
One qf <identifier> collect call is one launch, and several files in one call merge into it. With CircleCI parallelism:, each container writes its own report and one later job uploads them all with --shard. A job that names another in requires: starts only when that job succeeds, so two settings keep the upload when tests fail: a teardown on the test step saves each container's report to the workspace when the job ends, and test: [success, failed] under requires: starts the upload job after a failed test job too.
version: 2.1
jobs:
test:
docker:
- image: cimg/python:3.12
parallelism: 2
steps:
- checkout
- run:
name: Run tests
command: |
mkdir -p test-results
pytest $(circleci tests glob "tests/**/test_*.py" | circleci tests split) --junitxml=test-results/shard-$CIRCLE_NODE_INDEX.xml
teardown: # runs when the job ends, even when the tests fail
- persist_to_workspace:
root: .
paths:
- test-results/shard-*.xml
upload:
docker:
- image: cimg/python:3.12
environment:
QF_ENVIRONMENT: staging # must be an environment that exists in the project
steps:
- attach_workspace:
at: .
- run:
name: Upload results to Qualflare
command: |
VERSION=0.4.0
curl -L "https://github.com/qualflare/qualflare-cli/releases/download/v${VERSION}/qf_${VERSION}_linux_amd64.tar.gz" -o qf.tar.gz
tar -xzf qf.tar.gz && sudo mv qf /usr/local/bin/
export QF_BRANCH="$CIRCLE_BRANCH" QF_COMMIT="$CIRCLE_SHA1"
qf login ci --force
qf ci collect "test-results/*.xml" --shard
workflows:
test:
jobs:
- test
- upload:
requires:
- test: [success, failed] # upload whether the tests passed or failedcircleci tests split gives each container a share of the test files, and $CIRCLE_NODE_INDEX gives each container's report its own name. qf <identifier> collect exits 0 when the upload succeeds, so the test job still marks the workflow failed when tests fail, and a failed upload fails the upload job. See Sharded runs.
Pin the qf version
The example installs qf 0.4.0 through VERSION=0.4.0. To pin another release, set VERSION to a version from the qf releases page.
Troubleshooting
Error: token cannot be empty means QF_TOKEN didn't reach the job: attach the context that holds it, and note that a build of a pull request from a fork gets no project variables unless the project allows it. Launches with no branch: keep the export line in the upload step. More fixes: Troubleshooting on CI/CD integration.
Related
- Guides: CI/CD integration
- Reference: Upload results with qf collect · Configuration
- CLI/API: Access Tokens · Rotate the CI access token
Azure Pipelines
Upload test results from Azure Pipelines to Qualflare with the qf CLI: azure-pipelines.yml steps, a secret QF_TOKEN variable, and setting QF_BRANCH.
Bitbucket Pipelines
Upload test results from Bitbucket Pipelines to Qualflare with the qf CLI: an after-script that runs when tests fail and a secured QF_TOKEN repository variable.