Espresso reporter

com.qualflare:qualflare-espresso is the Qualflare reporter for Espresso: it runs inside the instrumented app and writes each test's results, with retries, steps and screenshots, to the folder Gradle pulls from the device, and qf <identifier> collect uploads them as one launch.

Before you begin

• The Qualflare CLI, qf: Install the CLI.
• A saved login: qf login <identifier> with a project access token whose creator still holds the project role that uploads need Editor+ · project (requires the Editor role or higher in the project) (Authenticate with qf login).
• The Identifier of the environment the results belong to (Environments).

Install

// app/build.gradle.kts
dependencies {
    androidTestImplementation("com.qualflare:qualflare-espresso:0.1.0")
}

android {
    defaultConfig {
        testInstrumentationRunnerArguments["listener"] =
            "com.qualflare.espresso.QualflareRunListener"
    }
}

JUnit 4 has no ServiceLoader hook, so the listener argument is required; it works with your existing runner, Hilt's included. The reporter adds nothing to your test classpath.

Configure

Set options as instrumentation arguments, in Gradle or for one run:

testInstrumentationRunnerArguments["qualflare.environment"] = "staging"

./gradlew connectedAndroidTest -Pandroid.testInstrumentationRunnerArguments.qualflare.environment=staging

Every reporter shares the other settings: see Reporter configuration.

Set qualflare.environment to the environment's Identifier: a name such as Staging fails the upload with HTTP 400 (Environment).

Run and upload

./gradlew connectedAndroidTest
qf <identifier> collect "app/build/outputs/connected_android_test_additional_output/*/connected/*/qualflare-espresso-*.json"

Gradle nests the reports in a folder per variant and device, and qf reads only the files directly inside a folder, so pass the files with a quoted glob.

Below API 29, Gradle passes no output folder. The reporter writes into the app's files folder and prints the commands to pull it, including the flag that keeps the app installed after the run.

Run in CI

- name: Upload results to Qualflare
  if: ${{ !cancelled() }}
  env:
    QF_TOKEN: ${{ secrets.QF_TOKEN }}
    QF_BRANCH: ${{ github.head_ref || github.ref_name }}
    QF_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
  run: |
    npm install -g @qualflare/cli
    qf login ci --force
    qf ci collect "app/build/outputs/connected_android_test_additional_output/*/connected/*/qualflare-espresso-*.json"

if: ${{ !cancelled() }} uploads the results when tests fail, and QF_BRANCH and QF_COMMIT record a pull request's own branch and commit (Pull request branches). For GitLab CI/CD, Jenkins and Azure Pipelines, see Run the upload when tests fail.

Sharding and retries

A rerun test is one test with every attempt, marked flaky when a rerun passed. The glob collects every device's reports into one launch. See Merge sharded CI jobs into one launch.

What gets uploaded

• Status: passed, failed, skipped, timeout or error.
• Retries.
• Named, nested steps with parameters.
• Screenshots on demand, or on failure with QualflareRule.
• Attachments: bytes, a File or a Bitmap.
• Labels, tags, links, priority and description.

Screenshot on failure:

private final ActivityScenarioRule<LoginActivity> activity =
        new ActivityScenarioRule<>(LoginActivity.class);

@Rule
public final RuleChain rules = RuleChain.outerRule(activity).around(new QualflareRule());

Put QualflareRule inside the rule that launches the activity. Reversed, the activity is already closed and the screenshot shows the launcher.

A test writes its own metadata:

@Test
public void signsIn() {
    Qualflare.label("team", "identity");
    Qualflare.tag("smoke");
}

qf doesn't read the branch, commit, language or milestone from the report: set them on the upload (Branch, commit, language and milestone).

Troubleshooting

• directory "<dir>" contains no .json report files: you passed the output folder; pass the quoted glob above.
• To see what the reporter did on the device, run adb logcat -s QualflareEspresso.
• environment "Staging" does not exist in this project: use the environment's Identifier, or pass --environment on the upload (Environment).
• Screenshots can't be downloaded: qf is older than 0.1.24. Upgrade qf.

Related

• Guides: Reporters · Upload results with qf collect · Sharded runs and result directories · Upload screenshots, videos and traces
• Reference: Reporter configuration · Espresso format
• More: Espresso test reporting in Qualflare