TestNG reporter

com.qualflare:qualflare-testng is the Qualflare reporter for TestNG 7.4.0 or later on Java 11 or later: it registers itself, writes each run's results, with retries, steps and attachments, to qualflare-results, and qf <identifier> collect uploads the folder as one launch.

Before you begin

• The Qualflare CLI, qf: Install the CLI.
• A saved login: qf login <identifier> with a project access token whose creator still holds the project role that uploads need Editor+ · project (requires the Editor role or higher in the project) (Authenticate with qf login).
• The Identifier of the environment the results belong to (Environments).

Install

Add the test dependency. The listener registers itself through TestNG's ServiceLoader: no @Listeners, no -listener flag, no testng.xml edit.

<dependency>
  <groupId>com.qualflare</groupId>
  <artifactId>qualflare-testng</artifactId>
  <version>0.1.0</version>
  <scope>test</scope>
</dependency>

With Gradle (build.gradle.kts):

testImplementation("com.qualflare:qualflare-testng:0.1.0")

Configure

Set options as qualflare.* system properties, or QUALFLARE_* variables. In Maven Surefire:

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <systemPropertyVariables>
      <qualflare.environment>staging</qualflare.environment>
    </systemPropertyVariables>
  </configuration>
</plugin>

In Gradle:

tasks.test {
    systemProperty("qualflare.environment", "staging")
}

qualflare.enabled is read when the report is written, so setting it in a @BeforeSuite works.

Every reporter shares the other settings: see Reporter configuration.

Set qualflare.environment to the environment's Identifier: a name such as Staging fails the upload with HTTP 400 (Environment).

Run and upload

mvn test
qf <identifier> collect ./qualflare-results

Each JVM writes its own file, and qf merges the folder into one launch.

Run in CI

- name: Upload results to Qualflare
  if: ${{ !cancelled() }}
  env:
    QF_TOKEN: ${{ secrets.QF_TOKEN }}
    QF_BRANCH: ${{ github.head_ref || github.ref_name }}
    QF_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
  run: |
    npm install -g @qualflare/cli
    qf login ci --force
    qf ci collect ./qualflare-results

if: ${{ !cancelled() }} uploads the results when tests fail, and QF_BRANCH and QF_COMMIT record a pull request's own branch and commit (Pull request branches). For GitLab CI/CD, Jenkins and Azure Pipelines, see Run the upload when tests fail.

Sharding and retries

Retries come from IRetryAnalyzer, in the same JVM: one test with every attempt, flaky when a retry passed. TestNG reports a retried failure as a skip with wasRetried() set; the reporter reads that flag, so the attempts aren't skips.

Maven Surefire's rerunFailingTestsCount doesn't apply to TestNG.

For shards that run as separate CI jobs, see Merge sharded CI jobs into one launch.

What gets uploaded

• Each test's status, with a timeout distinct from a failure, and its per-attempt history
• Nested steps and attachments
• Labels, tags, priority and parameters

import com.qualflare.testng.Qualflare;
import org.testng.annotations.Test;

public class CheckoutTest {
    @Test
    public void checksOut() {
        Qualflare.label("feature", "checkout");
        Qualflare.step("add to cart", () -> {
            Qualflare.maskedParameter("token");
        });
    }
}

The API finds the running test itself: no annotation. A call with no test in scope, a @BeforeMethod included, is dropped with a warning.

Troubleshooting

• testng-results.xml isn't a format qf reads; use the reporter, or JUnit XML with --format testng.
• A launch has results from an earlier run: the reporter writes no run ID, so qf uploads every report in qualflare-results. Empty the folder before each run (Run ID).
• environment "Staging" does not exist in this project: use the environment's Identifier, or pass --environment on the upload (Environment).
• Screenshots can't be downloaded: qf is older than 0.1.24. Upgrade qf.

Related

• Guides: Reporters · Upload results with qf collect · Sharded runs and result directories · Upload screenshots, videos and traces
• Reference: Reporter configuration · TestNG format
• More: TestNG test reporting in Qualflare