Overview dashboard

In Qualflare, the Overview is the one fixed dashboard each project has: six KPI tiles and eleven charts that show success rate, flakiness, durations and defects for the last 7, 15 or 28 days, one point per day.

How the Overview works

The Overview is each project's one dashboard. Open it from Overview at the top of the project sidebar. The layout is fixed.

Until the project's first launch arrives, the Overview shows a setup guide. It switches to the dashboard by itself when the first run comes in. See Collect Test Results.

Filters

Each range covers whole days in UTC, including today. Every chart shows one point per day.

Data freshness

The Updated label next to the filters shows how long ago Qualflare last collected trends, and the page checks it every minute. Trends are collected in 5-minute steps, so launch, case run and defect data reach the charts up to about 7 minutes after a launch finishes. The Automation tile and the Case Automation chart are counted once a day, so their newest point is the end of the previous UTC day. The page refreshes the charts by itself, and there is no refresh setting.

The label's age is the same for every project: it says when collection last ran, not when your project last had data. If it turns orange with a warning icon, collection is 20 minutes or more behind. Trends have not been collected yet means collection has never run, and Freshness unknown means Qualflare couldn't read the age, so the charts may be out of date.

KPI tiles

The KPI tiles are the row of six at the top. Each shows the latest day's value, a small trend line for the range, and the change against the daily average of the previous period of the same length.

A rising Success Rate or Automation is good. A rising Flake Rate, Launch p90 or Defects Opened is not. Launches has no good or bad direction.

A case run counts as flaky when it passed after a failed attempt. If your reporter sends no attempt history, Qualflare uses the reporter's own flaky flag. Flaky test detection is separate: it flags test cases whose results flip across recent runs, using the settings on the project's Quality settings page.

A launch's own Success Rate, on its Overview tab, divides passed case runs by its passed, failed, skipped, error, timeout and aborted ones, so skipped, error, timeout and aborted results lower it. The dashboard leaves those out, so its Success Rate can be higher. See Pass Rate Calculation.

Chart groups

The Overview's eleven charts sit in three chart groups, each answering one question. Hover over a chart to see its values for a day. An empty chart says what data it's waiting for.

Reliability

Can a green run be believed?

Performance

Is testing holding the pipeline up?

Quality

Are we getting better?

Quality targets

Quality targets give the Overview something to measure against. Set them in the project's Settings > Quality, under Quality Targets:

• Target Success Rate, Target Flake Rate and Target Automation Rate: 0 to 100%
• Target Launch P90 Duration: in seconds, up to 86,400 in the web app

Leave a field empty, or set it to 0, for no target. The Success Rate, Flake Rate, Automation and Launch p90 tiles compare against their targets, and the Flakiness and Launch Durations charts draw a target line. Defects Opened and Launches have no target.

Editor+ · workspace (requires the Editor role or higher in the workspace)

Saving these settings needs the Editor role or higher in the workspace. A project role alone isn't enough. In the web app today, you also need the Editor role or higher in this project. See Quality Settings.

Read trends

Reading trends on the Overview means following the Success Rate and Flake Rate across the range, not one day's value:

• Improving quality: Success Rate trending upward over time, with fewer dips or fluctuations, and Flake Rate trending downward.
• Quality regression: Success Rate declining across several days, new lows that don't recover, and failures rising across unrelated tests.
• Stable quality: Success Rate holding within a small range, with no significant upward or downward trend.
• Volatility: large swings from day to day, often a sign of flaky tests or an unstable environment. Compare environments in Success Rate by Environment.

Set quality targets

Setting quality targets takes three steps on the project's Quality settings page.

Track defect trends

Tracking defect trends starts in the Quality group: the Defects, Defects by Severity and Defects by Category charts, and the Defects Opened tile.

To list the defects behind the numbers, click Defects under Analysis in the project sidebar, and filter by Severity, Status, Category or Environment. The list shows the oldest defects first by default. Click Date Created to see the newest first.

• Healthy: Closed keeps pace with Opened, and few defects are Critical or High.
• Concerning: Opened stays above Closed day after day, Critical and High defects stay open, or one category keeps growing, for example Timeout or Environment error.

See Manage Defects.

Find and fix flaky tests

Finding and fixing flaky tests starts with flaky test detection, which flags test cases whose result flips between pass and fail across launches All plans (available on Starter, Core, Scale, Enterprise, Open Source).

How Qualflare flags a flaky test

Qualflare checks a test case again each time you upload a launch that contains it. It takes the case's most recent results, one per launch, and flags the case as flaky when all of these are true:

• The case has at least Min Executions results within the Analysis Window
• It both passed and failed. Error, timeout and aborted results count as failures. Skipped and pending results are ignored.
• The less common result (pass or fail) makes up at least the Minority Threshold of those results

The flaky score is that share of the less common result, so it is never above 50%. A score close to 50% means the test passes and fails about equally often.

A single case run that passed only after a retry is also marked flaky. Its badge shows the retry count.

The help text on the Quality settings page states this rule the other way round; Qualflare flags a case when the less common result reaches the threshold, as described here.

Find and act on flaky tests

1. Find flagged cases: open Browse in your project (Test Suites) and turn on the Flaky filter to list only flagged cases.
2. Inspect a case: open the test case. A flaky case shows a badge with its flaky score. The Overview tab shows how many of its recent runs were flaky, and the History tab lists each run, including runs that passed only after a retry.
3. Check AI Insights: open AI Insights Core, Scale, Enterprise (available on Core, Scale, Enterprise) and select the Flaky tab to see flagged cases. See AI Features.
4. Follow the defect: when a case is flagged, Qualflare opens a defect for it with the flaky category (shown as Flakiness, and as Flaky Test in the Category filter). Qualflare closes that defect when the case stops being flaky.
5. Tune detection: open Settings > Quality in your project. Under Flaky Test Detection, change Analysis Window, Min Executions or Minority Threshold, then click Save. See Quality Settings for the defaults and ranges.
6. Act on a flaky test: use the patterns below to find the cause, then fix the test or the code. If the test is obsolete, archive the case in Browse. Archived cases are left out of flaky detection.

Flaky test patterns

• Environment-specific: passes on one environment and fails on another, or fails only in production, which points to configuration differences.
• Timing-dependent: passes alone and fails in parallel, or times out now and then, from race conditions or delays.
• Data-dependent: fails when specific test data exists, or depends on the execution order, from tests sharing state or data.
• Test code issues: hard-coded timestamps or dates, brittle selectors or locators, or missing cleanup between runs.

Fix them in this order:

• First: cases with a flaky score close to 50%, which pass and fail about equally often, and critical-path tests that block releases.
• Then: cases just above the Minority Threshold, and last, runs that pass only after a retry while the case itself isn't flagged.

Milestone health

Health on a milestone is a bar of the case results in the most recent run of each suite across its linked launches, and its Overview tab shows the Pass Rate. Milestones have no success criteria or at-risk status; to check open defects before a release, filter Defects by Status and Severity.

See How Milestone Health Is Calculated, Milestones and Check Release Readiness.

Get notified

Qualflare has no pass-rate alerts. To hear about changes:

• Email and in-app notifications: In the project's Settings, click Notifications. The email settings are Cluster Notifications, Defect Notifications, Team Assignment Notifications and Import finished. The in-app settings are Flaky Case Insights, Low-Value Case Insights and Redundant Case Insights.
• Chat and other tools: Click Integrations in the project sidebar, connect a tool such as Slack, and choose the events it receives. See Slack. To call your own endpoint, add a webhook on the Webhooks tab of the project's Settings. See Webhooks.

Limits and plan availability

The Overview has these limits and plan rules:

• One fixed Overview per project. You can't add, move or remove charts, create other dashboards, or share, embed or export it.
• Ranges of 7, 15 or 28 whole UTC days only. There is no custom date range.
• Environment is the only filter. There is no suite, branch or framework filter.
• Every plan has the Overview and quality targets. Flaky test detection and AI Insights are as the facts table shows.

Related

• Guides: Reporting Test Results · Execute Tests · Manage Defects · Collaborate with Your Team
• Reference: Launches · Environments · Project Settings · Defects · Milestones reference · Search & Filter · Glossary
• CLI: Collect Test Results