Maestro reporter

qualflare-maestro is the Qualflare reporter for Maestro 2.6.0 or later: it runs maestro test and writes a step for every command, with screenshots and tags, to qualflare-results, and qf <identifier> collect uploads the folder as one launch.

Before you begin

• The Qualflare CLI, qf: Install the CLI.
• A saved login: qf login <identifier> with a project access token whose creator still holds the project role that uploads need Editor+ · project (requires the Editor role or higher in the project) (Authenticate with qf login).
• The Identifier of the environment the results belong to (Environments).

Install

brew install qualflare/tap/qualflare-maestro
# or
go install github.com/Qualflare/qualflare-maestro/cmd/qualflare-maestro@latest

Binaries for macOS, Linux and Windows are on the releases page.

Configure

Reporter flags go before --, and Maestro's after it:

qualflare-maestro -environment staging -- maestro test --env USER=ci .maestro/

The reporter sets --format, --output, --debug-output and --flatten-debug-output itself; passing one exits with code 2.

Every reporter shares the other settings: see Reporter configuration.

Set -environment to the environment's Identifier: a name such as Staging fails the upload with HTTP 400 (Environment).

Run and upload

qualflare-maestro -- maestro test .maestro/
qf <identifier> collect ./qualflare-results

qualflare-maestro .maestro/ is short for the first line. It returns Maestro's exit code, so it replaces maestro test in CI.

Run in CI

- name: Run Maestro flows
  run: qualflare-maestro -- maestro test .maestro/

- name: Upload results to Qualflare
  if: ${{ !cancelled() }}
  env:
    QF_TOKEN: ${{ secrets.QF_TOKEN }}
    QF_BRANCH: ${{ github.head_ref || github.ref_name }}
    QF_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
  run: |
    npm install -g @qualflare/cli
    qf login ci --force
    qf ci collect ./qualflare-results

if: ${{ !cancelled() }} uploads the results when tests fail, and QF_BRANCH and QF_COMMIT record a pull request's own branch and commit (Pull request branches). For GitLab CI/CD, Jenkins and Azure Pipelines, see Run the upload when tests fail.

Sharding and retries

Every shard must share one run ID (-run-id, detected from the CI run on GitHub Actions, GitLab CI, CircleCI and Jenkins); set -shard-index per shard. See Merge sharded CI jobs into one launch.

What gets uploaded

• A test per flow, named by its path and name.
• A step per command, named from the YAML, nested under runFlow, repeat and retry on Maestro 2.10 or later.
• Screenshots of failed steps, and of warned ones on Maestro 2.10 or later.
• Durations from the commands.
• The flow's tags.
• When Maestro exits without results, an error test named [unattributed failure] with the end of Maestro's log.

Flows set metadata in properties:

appId: com.example.app
name: Checkout with a saved card
properties:
  qualflare.priority: high
  qualflare.description: Pays with the first saved card and checks the receipt.
  qualflare.link.issue: https://example.atlassian.net/browse/PAY-142
  qualflare.link.custom.runbook: https://wiki.example.com/payments
  qualflare.label.team: payments

• qualflare.priority: low, medium, high or critical.
• qualflare.description: the description.
• qualflare.link.<issue|tms|custom>[.<name>]: a link, optionally named.
• qualflare.label.<name>: a label.
• Anything else: a test property.

Step names keep ${PASSWORD}-style placeholders, and values passed with --env or MAESTRO_* variables are replaced by ${NAME} in error messages.

qf doesn't read the branch, commit, language or milestone from the report: set them on the upload (Branch, commit, language and milestone).

Troubleshooting

• Exit 2: a usage error, or a flag the reporter sets.
• Exit 127: maestro wasn't found; set QUALFLARE_MAESTRO_BIN.
• Exit 1: Maestro couldn't start, or the report couldn't be written.
• An [unattributed failure] test: invalid YAML, or no device.
• environment "Staging" does not exist in this project: use the environment's Identifier, or pass --environment on the upload (Environment).
• Screenshots can't be downloaded: qf is older than 0.1.24. Upgrade qf.

Related

• Guides: Reporters · Upload results with qf collect · Sharded runs and result directories · Upload screenshots, videos and traces
• Reference: Reporter configuration · Maestro format
• More: Maestro test reporting in Qualflare