Detect and fix flaky tests
Find flaky tests in Qualflare: how a case is flagged and scored, the Quality settings that tune it, the defect it opens, and how to triage and fix it.
Last updated
To find flaky tests in Qualflare, upload results from every CI run: Qualflare scores each test case over its last 25 launches by default and flags it once it has 10 results there and the less common result, pass or fail, reaches 30%.
Before you begin
- Flaky test detection: All plans (available on Starter, Core, Scale, Enterprise, Open Source).
- Results uploaded with
qf <identifier> collector a Qualflare reporter: a case is checked when an upload contains it. - To catch flaky runs too, let your test framework retry a test that fails and keep the attempts in its report, as Playwright and Cypress do with retries.
- To change the settings: Editor+ · workspace (requires the Editor role or higher in the workspace), and in the web app today also Editor+ · project (requires the Editor role or higher in the project).
Flaky runs and flaky cases
Qualflare marks two kinds of flakiness:
- A flaky run is one case run that passed after a failed attempt. Its badge shows the retry count.
- A flaky case is a test case whose result flips across launches. Its badge shows the flaky score.
The Overview's Flake Rate counts flaky runs (KPI tiles).
How Qualflare flags a flaky test case
Qualflare checks a test case again each time you upload a launch that contains it. It takes the case's most recent results, one per launch, and flags the case as flaky when all of these are true:
- The case has at least Min Executions results within the Analysis Window
- It both passed and failed. Error, timeout and aborted results count as failures. Skipped and pending results are ignored.
- The less common result (pass or fail) makes up at least the Minority Threshold of those results
The flaky score is that share of the less common result, so it is never above 50%. A score close to 50% means the test passes and fails about equally often.
A single case run that passed only after a retry is also marked flaky. Its badge shows the retry count.
Results of manual runs of the case count too. Archived cases are never checked.
Where flagged cases show up
A flagged case shows up in these places:
| Where | What you see |
|---|---|
| Browse | The Flaky filter lists flagged cases (Test suites) |
| Test case | A Flaky badge with the score; Overview counts the recent flaky runs and History lists each run (Test cases) |
| Launch | The Flaky Tests count in the launch header |
| Overview dashboard | The Flake Rate tile and the Flakiness and Retry Burden charts |
| AI Insights | The Flaky tab, Core, Scale, Enterprise (available on Core, Scale, Enterprise) (AI Insights) |
| Defects | A defect named Flaky detection: <case name>, severity Medium, category Flakiness (Flaky Test in the filter) |
| Notifications | An in-app notice of new flaky tests, Core, Scale, Enterprise (available on Core, Scale, Enterprise); turn it off with Flaky Case Insights in the project's Settings > Notifications |
| Integrations | The Flaky Test Detected event, sent each time Qualflare opens a flaky defect for a case, for the integration rules that subscribe to it, such as Slack and Jira (Integrations) |
Find and fix flaky tests
List the flagged cases
Open Browse and turn on the Flaky filter.
Read the case's history
Open the case. Read its score on the badge, then its History tab, where runs that passed only after a retry are marked.
Check the defect Qualflare opened
On Defects, filter Category to Flaky Test. Qualflare closes that defect when the case stops being flaky; if the case is flagged again later, a new defect opens.
Find the cause
Use the flaky test patterns, then fix the test or the code.
Retire an obsolete test
Archive the case in Browse. Archived cases aren't checked.
Tune detection
To tune flaky detection, open the project's Settings > Quality. Under Flaky Test Detection, change Analysis Window, Min Executions or Minority Threshold, and click Save. The defaults and ranges are in Quality Settings.
Min Executions can't be higher than the Analysis Window. New values apply from each case's next check.
The help text on the Quality settings page states this rule the other way round; Qualflare flags a case when the less common result reaches the threshold, as described here.
Flaky test patterns
- Environment-specific: passes on one environment and fails on another, or fails only in production, which points to configuration differences.
- Timing-dependent: passes alone and fails in parallel, or times out now and then, from race conditions or delays.
- Data-dependent: fails when specific test data exists, or depends on the execution order, from tests sharing state or data.
- Test code issues: hard-coded timestamps or dates, brittle selectors or locators, or missing cleanup between runs.
Fix them in this order:
- First: cases with a flaky score close to 50%, which pass and fail about equally often, and critical-path tests that block releases.
- Then: cases just above the Minority Threshold, and last, runs that pass only after a retry while the case itself isn't flagged.
Set a flake rate target
To set a flake rate target, enter Target Flake Rate under Quality Targets in the project's Settings > Quality and click Save; it draws a target line on Flakiness (Quality targets).
What Qualflare doesn't do
Qualflare doesn't quarantine, skip or re-run a test. To quarantine one, use your test runner.
Verify
After the next upload, the Flaky filter in Browse lists the cases you expect, each with its score.
Troubleshoot
| Problem | Cause and fix |
|---|---|
| A case that fails now and then isn't flagged | It has fewer than Min Executions results in the window, or the less common result is under Minority Threshold |
| No run is ever marked flaky | Your report has no retry data. Turn on retries in your test framework |
| You closed the flaky defect, or marked it False positive, and a new one opened | The case is still flagged, so the next upload that contains it opens another. Fix the test, or archive the case if it's obsolete |
| The Quality page says the minority "must be below" the threshold | The help text is reversed; the rule on this page is the one Qualflare applies |
Next steps
Overview dashboard
Read the Qualflare Overview dashboard: success rate, flake rate, launch durations and defect trends per project, flaky test detection and quality targets.
AI in Qualflare
Qualflare's AI features: AI Insights, Failure Clusters, AI Launch Analysis, Quo and the AI filter, with the project AI switch and per-plan usage limits.