# Detect and fix flaky tests Source: https://docs.qualflare.com/docs/analytics/flaky-tests ยท Last updated: 2026-10-28 Find flaky tests in Qualflare: how a case is flagged and scored, the Quality settings that tune it, the defect it opens, and how to triage and fix it. To find flaky tests in Qualflare, upload results from every CI run: Qualflare scores each test case over its last 25 launches by default and flags it once it has 10 results there and the less common result, pass or fail, reaches 30%. ## Before you begin * Flaky test detection: (available on Starter, Core, Scale, Enterprise, Open Source). * Results uploaded with `qf collect` or a Qualflare reporter: a case is checked when an upload contains it. * To catch flaky runs too, let your test framework retry a test that fails and keep the attempts in its report, as Playwright and Cypress do with retries. * To change the settings: (requires the Editor role or higher in the workspace), and in the web app today also (requires the Editor role or higher in the project). ## Flaky runs and flaky cases Qualflare marks two kinds of flakiness: * A **flaky run** is one case run that passed after a failed attempt. Its badge shows the retry count. * A **flaky case** is a test case whose result flips across launches. Its badge shows the flaky score. The Overview's **Flake Rate** counts flaky runs ([KPI tiles](https://docs.qualflare.com/docs/analytics#kpi-tiles)). ## How Qualflare flags a flaky test case Qualflare checks a test case again each time you upload a launch that contains it. It takes the case's most recent results, one per launch, and flags the case as flaky when all of these are true: * The case has at least **Min Executions** results within the **Analysis Window** * It both passed and failed. Error, timeout and aborted results count as failures. Skipped and pending results are ignored. * The less common result (pass or fail) makes up at least the **Minority Threshold** of those results The **flaky score** is that share of the less common result, so it is never above 50%. A score close to 50% means the test passes and fails about equally often. A single case run that passed only after a retry is also marked flaky. Its badge shows the retry count. Results of manual runs of the case count too. Archived cases are never checked. ## Where flagged cases show up A flagged case shows up in these places: | Where | What you see | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Browse** | The **Flaky** filter lists flagged cases ([Test suites](https://docs.qualflare.com/docs/test-management/test-suites)) | | Test case | A Flaky badge with the score; **Overview** counts the recent flaky runs and **History** lists each run ([Test cases](https://docs.qualflare.com/docs/test-management/test-cases)) | | Launch | The **Flaky Tests** count in the launch header | | **Overview** dashboard | The **Flake Rate** tile and the **Flakiness** and **Retry Burden** charts | | **AI Insights** | The **Flaky** tab, (available on Core, Scale, Enterprise) ([AI Insights](https://docs.qualflare.com/docs/ai/insights)) | | **Defects** | A defect named `Flaky detection: `, severity Medium, category **Flakiness** (**Flaky Test** in the filter) | | Notifications | An in-app notice of new flaky tests, (available on Core, Scale, Enterprise); turn it off with **Flaky Case Insights** in the project's **Settings** > **Notifications** | | Integrations | The **Flaky Test Detected** event, sent each time Qualflare opens a flaky defect for a case, for the integration rules that subscribe to it, such as [Slack](https://docs.qualflare.com/docs/integrations/slack) and [Jira](https://docs.qualflare.com/docs/integrations/jira) ([Integrations](https://docs.qualflare.com/docs/integrations)) | *Figure: Turn on the Flaky filter in Browse to list only the cases Qualflare has flagged.* *Figure: A launch shows how many of its case runs were flaky.* *Figure: The Flaky tab in AI Insights lists flagged cases on plans that include AI Insights.* ## Find and fix flaky tests ### Step 1: List the flagged cases Open **Browse** and turn on the **Flaky** filter. ### Step 2: Read the case's history Open the case. Read its score on the badge, then its **History** tab, where runs that passed only after a retry are marked. ### Step 3: Check the defect Qualflare opened On **Defects**, filter **Category** to **Flaky Test**. Qualflare closes that defect when the case stops being flaky; if the case is flagged again later, a new defect opens. ### Step 4: Find the cause Use the [flaky test patterns](https://docs.qualflare.com/docs/analytics/flaky-tests#flaky-test-patterns), then fix the test or the code. ### Step 5: Retire an obsolete test Archive the case in **Browse**. Archived cases aren't checked. ## Tune detection To tune flaky detection, open the project's **Settings** > **Quality**. Under **Flaky Test Detection**, change **Analysis Window**, **Min Executions** or **Minority Threshold**, and click **Save**. The defaults and ranges are in [Quality Settings](https://docs.qualflare.com/docs/administration/project-settings#quality-settings). Min Executions can't be higher than the Analysis Window. New values apply from each case's next check. The help text on the **Quality** settings page states this rule the other way round; Qualflare flags a case when the less common result reaches the threshold, as described here. *Figure: Tune flaky detection in the project's Quality settings; the defaults are 25, 10 and 30 percent.* ## Flaky test patterns * **Environment-specific:** passes on one environment and fails on another, or fails only in production, which points to configuration differences. * **Timing-dependent:** passes alone and fails in parallel, or times out now and then, from race conditions or delays. * **Data-dependent:** fails when specific test data exists, or depends on the execution order, from tests sharing state or data. * **Test code issues:** hard-coded timestamps or dates, brittle selectors or locators, or missing cleanup between runs. Fix them in this order: * **First:** cases with a flaky score close to 50%, which pass and fail about equally often, and critical-path tests that block releases. * **Then:** cases just above the **Minority Threshold**, and last, runs that pass only after a retry while the case itself isn't flagged. ## Set a flake rate target To set a flake rate target, enter **Target Flake Rate** under **Quality Targets** in the project's **Settings** > **Quality** and click **Save**; it draws a target line on **Flakiness** ([Quality targets](https://docs.qualflare.com/docs/analytics#quality-targets)). ## What Qualflare doesn't do Qualflare doesn't quarantine, skip or re-run a test. To quarantine one, use your test runner. ## Verify After the next upload, the **Flaky** filter in **Browse** lists the cases you expect, each with its score. ## Troubleshoot | Problem | Cause and fix | | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | | A case that fails now and then isn't flagged | It has fewer than **Min Executions** results in the window, or the less common result is under **Minority Threshold** | | No run is ever marked flaky | Your report has no retry data. Turn on retries in your test framework | | You closed the flaky defect, or marked it False positive, and a new one opened | The case is still flagged, so the next upload that contains it opens another. Fix the test, or archive the case if it's obsolete | | The **Quality** page says the minority "must be below" the threshold | The help text is reversed; the rule on this page is the one Qualflare applies | ## Next steps * [Overview dashboard](https://docs.qualflare.com/docs/analytics) * [AI Insights](https://docs.qualflare.com/docs/ai/insights) * [Quality Settings](https://docs.qualflare.com/docs/administration/project-settings#quality-settings) * [Create, triage and close defects](https://docs.qualflare.com/docs/defects/manage-defects) * [How uploaded results become suites and cases](https://docs.qualflare.com/docs/runs/automated-results)