pytest reporter

qualflare-pytest is the Qualflare reporter for pytest 7.0 or later on Python 3.9 or later: the plugin writes each run's results, with per-attempt retries, steps and attachments, to qualflare-results, and qf <identifier> collect uploads the folder as one launch.

Before you begin

• The Qualflare CLI, qf: Install the CLI.
• A saved login: qf login <identifier> with a project access token whose creator still holds the project role that uploads need Editor+ · project (requires the Editor role or higher in the project) (Authenticate with qf login).
• The Identifier of the environment the results belong to (Environments).

Install

pip install qualflare-pytest

The plugin registers itself when installed: there is no -p flag to add. qf is a separate binary, not a Python package: install it with Homebrew, npm or a release binary.

Configure

In pytest.ini:

[pytest]
qualflare_environment = staging
qualflare_output_dir = qualflare-results

or in pyproject.toml:

# pyproject.toml
[tool.pytest.ini_options]
qualflare_environment = "staging"

The plugin reads pytest.ini, pyproject.toml, setup.cfg and tox.ini.

Every reporter shares the other settings: see Reporter configuration.

Set qualflare_environment to the environment's Identifier: a name such as Staging fails the upload with HTTP 400 (Environment).

Run and upload

pytest
qf <identifier> collect ./qualflare-results

Run in CI

- name: Upload results to Qualflare
  if: ${{ !cancelled() }}
  env:
    QF_TOKEN: ${{ secrets.QF_TOKEN }}
    QF_BRANCH: ${{ github.head_ref || github.ref_name }}
    QF_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
  run: |
    npm install -g @qualflare/cli
    qf login ci --force
    qf ci collect ./qualflare-results

if: ${{ !cancelled() }} uploads the results when tests fail, and QF_BRANCH and QF_COMMIT record a pull request's own branch and commit (Pull request branches). For GitLab CI/CD, Jenkins and Azure Pipelines, see Run the upload when tests fail.

Sharding and retries

pytest -n auto works with no configuration: the controller writes one report for the whole run, and each test's xdist worker becomes its shard index.

pytest-rerunfailures is supported: every attempt's status and error is uploaded.

For shards that run as separate CI jobs, see Merge sharded CI jobs into one launch.

What gets uploaded

• Each test's status, with its per-attempt retry history and flakiness
• Steps and attachments
• Labels, links, tags, priority and parameters

Markers such as @pytest.mark.smoke become tags, and with pytest-bdd installed, Gherkin steps keep their keyword and location.

import os
from qualflare_pytest import qualflare

def test_checks_out():
    qualflare.label("feature", "checkout")

    with qualflare.step("add to cart"):
        qualflare.parameter("sku", "widget")
        qualflare.parameter("token", os.environ["TOKEN"], masked=True)

Troubleshooting

• xfail and xpass upload as skipped and passed; under xfail(strict=True) an unexpected pass reads as a pass.
• record_property is ignored: use qualflare.parameter() or qualflare.label().
• Captured stdout and stderr aren't attached: attach what you need.
• environment "Staging" does not exist in this project: use the environment's Identifier, or pass --environment on the upload (Environment).
• Screenshots can't be downloaded: qf is older than 0.1.24. Upgrade qf.

Related

• Guides: Reporters · Upload results with qf collect · Sharded runs and result directories · Upload screenshots, videos and traces
• Reference: Reporter configuration · pytest format
• More: pytest test reporting in Qualflare