Skip to content
The docs are now organized by product area. Old links still work. Report a broken link

CircleCI

Upload test results from CircleCI to Qualflare with the qf CLI: a config.yml step that runs when tests fail, a QF_TOKEN variable, and QF_BRANCH from CircleCI.

Last updated

To upload test results from CircleCI to Qualflare, add a step with when: always that runs qf <identifier> collect <report>, with the access token in a project environment variable named QF_TOKEN; qf reads no CircleCI variables, so set QF_BRANCH and QF_COMMIT yourself.

Before you begin

  • A project access token with Allow writes off, which you create in Project Settings > Access Tokens Editor+ · project (requires the Editor role or higher in the project). See Access Tokens.
  • A report file your test step writes, such as JUnit XML. The examples use ci as the identifier.

Store the token as an environment variable

  1. In CircleCI, open your project's Project Settings > Environment Variables
  2. Click Add Variable
  3. Name: QF_TOKEN
  4. Value: Your Qualflare access token
  5. Save the variable

A context that holds QF_TOKEN, attached to the job with context: in the workflow, works the same way; with parallel jobs, attach it to the upload job too.

Add the upload step

Add to your .circleci/config.yml:

version: 2.1

jobs:
  test:
    docker:
      - image: cimg/python:3.12
    environment:
      QF_ENVIRONMENT: staging  # must be an environment that exists in the project
    steps:
      - checkout
      - run:
          name: Run tests
          command: pytest --junitxml=results.xml
      - run:
          name: Upload results to Qualflare
          when: always  # upload even when the tests fail
          command: |
            VERSION=0.4.0
            curl -L "https://github.com/qualflare/qualflare-cli/releases/download/v${VERSION}/qf_${VERSION}_linux_amd64.tar.gz" -o qf.tar.gz
            tar -xzf qf.tar.gz && sudo mv qf /usr/local/bin/
            export QF_BRANCH="$CIRCLE_BRANCH" QF_COMMIT="$CIRCLE_SHA1"
            qf login ci --force
            qf ci collect results.xml

workflows:
  test:
    jobs:
      - test

The image needs curl, tar and sudo, which the cimg images have.

Branch and commit

qf reads no CircleCI variable, so the export line in the upload step sets QF_BRANCH and QF_COMMIT from CIRCLE_BRANCH and CIRCLE_SHA1. It runs in the shell because CircleCI doesn't expand variables under environment:. On a tag build CIRCLE_BRANCH is empty: qf skips an empty variable, and the local checkout decides. The Branch and commit detection table lists what qf reads on each CI.

Run the upload when tests fail

when: always on the upload step runs it after a failed test step. qf <identifier> collect exits 0 when the upload succeeds, so the failed test step still fails the job.

With when: always, the upload step runs after a failed test step, and the job stays failed.

Parallel jobs

One qf <identifier> collect call is one launch, and several files in one call merge into it. With CircleCI parallelism:, each container writes its own report and one later job uploads them all with --shard. A job that names another in requires: starts only when that job succeeds, so two settings keep the upload when tests fail: a teardown on the test step saves each container's report to the workspace when the job ends, and test: [success, failed] under requires: starts the upload job after a failed test job too.

version: 2.1

jobs:
  test:
    docker:
      - image: cimg/python:3.12
    parallelism: 2
    steps:
      - checkout
      - run:
          name: Run tests
          command: |
            mkdir -p test-results
            pytest $(circleci tests glob "tests/**/test_*.py" | circleci tests split) --junitxml=test-results/shard-$CIRCLE_NODE_INDEX.xml
          teardown:  # runs when the job ends, even when the tests fail
            - persist_to_workspace:
                root: .
                paths:
                  - test-results/shard-*.xml

  upload:
    docker:
      - image: cimg/python:3.12
    environment:
      QF_ENVIRONMENT: staging  # must be an environment that exists in the project
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Upload results to Qualflare
          command: |
            VERSION=0.4.0
            curl -L "https://github.com/qualflare/qualflare-cli/releases/download/v${VERSION}/qf_${VERSION}_linux_amd64.tar.gz" -o qf.tar.gz
            tar -xzf qf.tar.gz && sudo mv qf /usr/local/bin/
            export QF_BRANCH="$CIRCLE_BRANCH" QF_COMMIT="$CIRCLE_SHA1"
            qf login ci --force
            qf ci collect "test-results/*.xml" --shard

workflows:
  test:
    jobs:
      - test
      - upload:
          requires:
            - test: [success, failed]  # upload whether the tests passed or failed

circleci tests split gives each container a share of the test files, and $CIRCLE_NODE_INDEX gives each container's report its own name. qf <identifier> collect exits 0 when the upload succeeds, so the test job still marks the workflow failed when tests fail, and a failed upload fails the upload job. See Sharded runs.

Pin the qf version

The example installs qf 0.4.0 through VERSION=0.4.0. To pin another release, set VERSION to a version from the qf releases page.

Troubleshooting

Error: token cannot be empty means QF_TOKEN didn't reach the job: attach the context that holds it, and note that a build of a pull request from a fork gets no project variables unless the project allows it. Launches with no branch: keep the export line in the upload step. More fixes: Troubleshooting on CI/CD integration.