Run tests: manual and automated
Run tests in Qualflare by hand in the Test Wizard, or upload automated results with the qf CLI. Each run becomes a launch you can review, share and compare.
Last updated
In Qualflare, every test run becomes a launch: you run cases by hand in the Test Wizard, or a CI job uploads results with qf <identifier> collect, and each upload creates one launch.
Manual or automated
A manual run and an automated upload both create a launch, but they start, finish and count differently.
| Manual run | Automated upload | |
|---|---|---|
| How you start it | Start Launch on Launches, or Create Run on a test plan | qf <identifier> collect <report> in CI or locally, for any supported report, including the file a Qualflare reporter writes |
| Who | Editor+ · project (requires the Editor role or higher in the project) | A project access token (Access tokens) |
| What it records | One verdict per case | Every result in the report |
| Environment | Picked in the dialog or the wizard, then locked | --environment, then QF_ENVIRONMENT, then the report's value, else development |
| Until it's done | Pending until you click Finalize Launch | Complete as soon as the upload finishes |
| Afterwards | Nothing else runs | AI launch analysis, failure clustering when tests failed, Slack and webhook notifications, and the public report, where they're set up |
| Monthly Launches limit | Doesn't count | Each upload counts once |
Run tests by hand
To run tests by hand, click Start Launch on Launches and pick suites: every case in them is included. Qualflare then opens the Test Wizard, where you record one verdict per case; see Run manual tests.
Run a saved test plan
To run a saved test plan, click Create Run on the plan's row in Test Plans; see Run a test plan.
Upload automated results
Upload automated results with the Qualflare CLI from your CI/CD pipeline or your machine. You need:
qfinstalled (Install the CLI)- results in a supported format (JUnit, pytest, Jest, etc.)
- a project access token saved with
qf login(Authentication)
Save your token once under a short local name. qf asks for the token and hides what you type:
qf login myappThen upload your results file:
qf myapp collect results.xmlqf detects the format, parses the results and creates one launch, with one case run per test. You cannot name a launch from the CLI.
Set the environment
# Upload with environment metadata
qf myapp collect results.xml --environment staging--environment takes the Identifier of an environment that already exists in the project. New projects have development, staging, production and qa. Without the flag, qf uses QF_ENVIRONMENT, then the environment a native reporter wrote into its report, then development.
An identifier the project doesn't have fails the whole upload with HTTP 400; see Use environments from the CLI.
Add build metadata
# Upload with CI/CD metadata
qf myapp collect results.xml \
--environment staging \
--branch feature/new-auth \
--commit abc123defOn GitHub Actions, GitLab CI and Bitbucket Pipelines, qf reads the branch and commit from the CI variables, so you can leave out --branch and --commit there.
Check a report before you upload
# Validate your results file before uploading
qf validate results.xml
# Parse the file and print what would be sent, without uploading
qf myapp collect results.xml --dry-run --output json
# List the supported formats and their --format values
qf list-formatsFor every flag and format, see Upload results with qf collect. To upload from a pipeline, see CI/CD.
Review results
To review results, open the launch from Launches and read its Overview tab, then open the failed case runs on its Test Cases tab; see Analyze launch results. Group related failures across launches with Failure Clusters, and read the run's summary in AI Launch Analysis.
Statuses
A launch's status is Passed, Failed, Error, Skipped or Pending, and each case run has one of seven statuses. Both are listed in Launch statuses and Case run statuses.
In this section
Run manual tests
Start a launch from suites or a test plan and record a verdict for each case in the Test Wizard.
Launches
Launch and case run statuses, the pass rate, the launch detail, run history and shards.
Analyze launch results
Read a launch's success rate, open failed case runs and triage failures across launches.
Share and export results
Publish a public report, post to Slack or a webhook, and get launches as JSON or CTRF.
Environments
Where tests ran: the four default environments, how uploads use them and how to delete one.
Limits and plans
- Uploads count toward the workspace's monthly Launches limit, and manual runs don't. See Limits.
- Start Launch takes up to 100 suites.
- AI launch analysis has its own monthly allowance, and some plans have no cap on it. See AI usage limits and AI Launch Analysis.
Related
- Guides: Run manual tests · Analyze launch results · Share and export results
- Reference: Launches · Environments · Glossary
- CLI and API:
qf <identifier> launches list· qf collect · Publishing a report · Webhooks
Test Plans
A test plan in Qualflare is a saved selection of test cases you run by hand in the Test Wizard. Create a plan, pick its cases, run it and check its limits.
Run manual tests with the Test Wizard
Run manual tests in Qualflare: start a launch from suites or a test plan, then record Passed, Failed, Blocked or Skipped for each case in the Test Wizard.