Clean up flaky tests
Find tests that fail at random and fix or quarantine them.
Engineering · Software · On a schedule · Updated
Works with
- Slack
SlackHow do you clean up flaky tests?
To clean up flaky tests, set a Loop in Maito for Monday morning. It reads the last week of test runs on main, finds tests that both pass and fail on the same code, fixes the ones with a clear cause, runs each fix many times, and opens a pull request. It reports the ones that need a person.
How it's done today
Flaky tests get re-run until green.
What Maito does
- Read recent test runs
- Find tests that pass and fail on the same code
- Fix or propose quarantine
Example
Loop run: Monday 7:00 AM · Flaky test cleanup
3 flaky tests found in 140 runs on main. 2 fixed in PR #915, 1 needs a person. • checkout.spec, applies coupon after address: failed 9 of 140. A fixture wasn't reset after cart.spec. Fixed. • invoice-pdf.test, renders in Berlin: failed 6 of 140. It formatted dates in the machine's time zone. Fixed by pinning TZ. Each ran 50 times green. • sync.worker.test, retries after timeout: failed 4 of 140, always after a slow run. Catch: I didn't touch the sync worker test. The test waits on a real timer, but the code under test may send the retry twice when the timeout is close. That could be a real bug, not a test problem. PR #915 is not merged.
@Maito skip the sync test until Thursday so main stays green.
Skipped it in PR #915 with a note, so it can't be forgotten. I'll look at the retry code first if you want.
Maito (7:00 AM):
3 flaky tests found in 140 runs on main. 2 fixed in PR #915, 1 needs a person.
• checkout.spec, applies coupon after address: failed 9 of 140. A fixture wasn't reset after cart.spec. Fixed.
• invoice-pdf.test, renders in Berlin: failed 6 of 140. It formatted dates in the machine's time zone. Fixed by pinning TZ. Each ran 50 times green.
• sync.worker.test, retries after timeout: failed 4 of 140, always after a slow run.
Catch: I didn't touch the sync worker test. The test waits on a real timer, but the code under test may send the retry twice when the timeout is close. That could be a real bug, not a test problem.
PR #915 is not merged.
Alex (8:15 AM):
skip the sync test until Thursday so main stays green.
Maito (8:19 AM):
Skipped it in PR #915 with a note, so it can't be forgotten. I'll look at the retry code first if you want.
How to set it up
- Ask your agent
“Every Monday at 7, look at the last week of test runs on main, find tests that pass and fail on the same code, fix the ones you can in a pull request and tell #eng about the rest.”
What it won't do
It works from test history it can read. A flaky test that fails once in a thousand runs may need more data than a week provides.
Questions
It looks for the same test failing and passing on the same commit across runs, and counts how often it fails.
Only if you ask. It first tries to fix the cause, and it names any test it left alone.
It runs the fixed test repeatedly and reports the result in the pull request. Still, a rare failure can survive.
