Cucumber.js reporter

@qualflare/cucumberjs is the Qualflare formatter for Cucumber.js 10.8 or later: it writes each run's Feature and Scenario results, with Given/When/Then steps, retries, screenshots and videos, to ./qualflare-results, and qf <identifier> collect uploads the folder as one launch.

Before you begin

• The Qualflare CLI, qf: Install the CLI.
• A saved login: qf login <identifier> with a project access token whose creator still holds the project role that uploads need Editor+ · project (requires the Editor role or higher in the project) (Authenticate with qf login).
• The Identifier of the environment the results belong to (Environments).

Install

npm install --save-dev @qualflare/cucumberjs

Configure

In cucumber.json:

{
  "default": {
    "format": ["@qualflare/cucumberjs/formatter"],
    "formatOptions": {
      "environment": "staging"
    }
  }
}

Every reporter shares the other settings: see Reporter configuration.

Set environment to the environment's Identifier: a name such as Staging fails the upload with HTTP 400 (Environment).

Run and upload

npx cucumber-js
qf <identifier> collect ./qualflare-results

Screenshots, including the ones you pass to World.attach(), upload by default; add --upload-artifacts video for videos.

Run in CI

- name: Upload results to Qualflare
  if: ${{ !cancelled() }}
  env:
    QF_TOKEN: ${{ secrets.QF_TOKEN }}
    QF_BRANCH: ${{ github.head_ref || github.ref_name }}
    QF_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
  run: |
    npm install -g @qualflare/cli
    qf login ci --force
    qf ci collect ./qualflare-results

if: ${{ !cancelled() }} uploads the results when tests fail, and QF_BRANCH and QF_COMMIT record a pull request's own branch and commit (Pull request branches). For GitLab CI/CD, Jenkins and Azure Pipelines, see Run the upload when tests fail.

Sharding and retries

Run each shard with --shard "$SHARD_INDEX/$SHARD_TOTAL" into the same folder; the formatter reads the shard from the command line. The upload needs no --shard flag. See Merge sharded CI jobs into one launch.

Each attempt's status, duration and error is uploaded; steps, labels, tags and attachments come from the final attempt.

What gets uploaded

• Feature and Scenario status, with the history of each retried attempt.
• Screenshots and videos.
• Given/When/Then steps and Scenario Outline rows.
• Labels, links, tags and attachments.

A step definition sets its own metadata with the qualflare calls:

import { Given, When } from '@cucumber/cucumber';
import { qualflare } from '@qualflare/cucumberjs';

Given('a user with valid credentials', function () {
  qualflare.label('epic', 'Authentication');
  qualflare.tag('smoke');
});

When('they log in', async function () {
  await qualflare.step('fill in credentials', async () => {
    await this.page.fill('#email', 'user@example.com');
    await this.page.fill('#password', 'correct-horse-battery-staple');
  });
});

qf doesn't read the branch, commit, language or milestone from the report: set them on the upload (Branch, commit, language and milestone).

Troubleshooting

• Attachments from a passing BeforeAll or AfterAll hook are dropped; a failing one becomes its own test with them.
• environment "Staging" does not exist in this project: use the environment's Identifier, or pass --environment on the upload (Environment).
• Screenshots can't be downloaded: qf is older than 0.1.24. Upgrade qf.

Related

• Guides: Reporters · Upload results with qf collect · Sharded runs and result directories · Upload screenshots, videos and traces
• Reference: Reporter configuration · Cucumber format
• More: Cucumber.js test reporting in Qualflare