# Flaky Tests: Find, Fix or Quarantine Them Source: https://maito.app/workflows/clean-up-flaky-tests Maito reviews a week of test runs, finds tests that pass and fail on the same code, fixes the clear causes and opens a pull request for the rest. Find tests that fail at random and fix or quarantine them. [Engineering](https://maito.app/workflows/team/engineering) · [Software](https://maito.app/workflows/industry/software) · On a schedule · Updated October 19, 2026 Works with - Slack - Email ## How do you clean up flaky tests? To clean up flaky tests, set a Loop in Maito for Monday morning. It reads the last week of test runs on main, finds tests that both pass and fail on the same code, fixes the ones with a clear cause, runs each fix many times, and opens a pull request. It reports the ones that need a person. ## How it's done today Flaky tests get re-run until green. ## What Maito does 1. Read recent test runs 2. Find tests that pass and fail on the same code 3. Fix or propose quarantine ## Example Loop run: Monday 7:00 AM · Flaky test cleanup **Maito** (7:00 AM): 3 flaky tests found in 140 runs on main. 2 fixed in PR #915, 1 needs a person. • checkout.spec, applies coupon after address: failed 9 of 140. A fixture wasn't reset after cart.spec. Fixed. • invoice-pdf.test, renders in Berlin: failed 6 of 140. It formatted dates in the machine's time zone. Fixed by pinning TZ. Each ran 50 times green. • sync.worker.test, retries after timeout: failed 4 of 140, always after a slow run. **Catch:** I didn't touch the sync worker test. The test waits on a real timer, but the code under test may send the retry twice when the timeout is close. That could be a real bug, not a test problem. PR #915 is not merged. **Alex** (8:15 AM): skip the sync test until Thursday so main stays green. **Maito** (8:19 AM): Skipped it in PR #915 with a note, so it can't be forgotten. I'll look at the retry code first if you want. ## How to set it up On a schedule Clean up flaky tests Ask your agent “Every Monday at 7, look at the last week of test runs on main, find tests that pass and fail on the same code, fix the ones you can in a pull request and tell #eng about the rest.” ## What it won't do It works from test history it can read. A flaky test that fails once in a thousand runs may need more data than a week provides. ## Questions ### How does it know a test is flaky? It looks for the same test failing and passing on the same commit across runs, and counts how often it fails. ### Does it just disable flaky tests? Only if you ask. It first tries to fix the cause, and it names any test it left alone. ### Does it prove a fix works? It runs the fixed test repeatedly and reports the result in the pull request. Still, a rare failure can survive. ## More engineering workflows - [Turn a ticket into a tested pull request Hand over a well-scoped ticket and get back a pull request with tests. Engineering · When you ask Works with Linear, Jira.](https://maito.app/workflows/turn-a-ticket-into-a-tested-pull-request) - [Review a teammate's pull request Get a first review that catches bugs and missed cases before a person looks. Engineering · When something happens Works with Slack, Email.](https://maito.app/workflows/review-a-teammates-pull-request) - [Check a change for security problems Look at a change for injection, leaked secrets and missing permission checks. Engineering · When you ask Works with Slack, Email.](https://maito.app/workflows/check-a-change-for-security-problems) - [Write the changelog Twice a week, turn merged pull requests into a changelog people can read. Engineering · On a schedule Works with Slack, Email.](https://maito.app/workflows/write-the-changelog) - [Nudge stale pull requests Pull requests waiting too long get a polite reminder to the right reviewer. Engineering · On a schedule Works with Slack, Email.](https://maito.app/workflows/nudge-stale-pull-requests) - [File a bug from a Slack thread When a bug comes up in chat, mention Maito and it gets filed once, with details and no duplicates. Engineering · When you ask Works with Linear, Jira.](https://maito.app/workflows/file-a-bug-from-a-slack-thread) [See all workflows](https://maito.app/workflows) ## Put your AI teammate to work. Set it up in minutes. Start with this one, and let it earn the next one. [Put Maito to work](https://maito.app/login?returnTo=%2Fworkflows%2Fclean-up-flaky-tests)[Book a demo](mailto:support@maito.app?subject=Maito%20demo)