Nightly regression run
Your key flows get tested every night, with a report in the morning.
QA · Software · On a schedule · Updated
Works with
- Slack
SlackHow do you automate regression testing?
To automate regression testing, set a Loop in Maito for each night. It opens the latest build in a browser, runs your key flows such as signup, checkout and export with each user role, and posts a report in Slack by morning. For any failure it attaches a video and names the likely cause.
How it's done today
Regressions are found by customers.
What Maito does
- Each night, run the main flows
- Compare with the last good run
- Report what changed
Example
Loop run: Nightly 2:00 AM · Regression run
Nightly run on build 4.13-rc2 (commit 3be91a): 2 of 3 flows pass. • Signup: 12 of 12 steps pass • Checkout: 17 of 17 steps pass • Export: fails at step 4. The Export CSV button is missing for Viewers. Last fully green run: Oct 17. PR #887, Role-based toolbar, was merged Oct 18 at 4:40 PM. It's the likely cause. Catch: Admin and Editor still see the button. A run on one admin account would have passed. I tested all three roles. Attached: export-viewer-oct-19.mp4.
@Maito is this on production too?
No. Production runs 4.12, which predates #887. I checked with the same Viewer account and the button is there.
Maito (2:21 AM):
Nightly run on build 4.13-rc2 (commit 3be91a): 2 of 3 flows pass.
• Signup: 12 of 12 steps pass
• Checkout: 17 of 17 steps pass
• Export: fails at step 4. The Export CSV button is missing for Viewers.
Last fully green run: Oct 17. PR #887, Role-based toolbar, was merged Oct 18 at 4:40 PM. It's the likely cause.
Catch: Admin and Editor still see the button. A run on one admin account would have passed. I tested all three roles.
Attached: export-viewer-oct-19.mp4.
Mia (8:50 AM):
is this on production too?
Maito (8:57 AM):
No. Production runs 4.12, which predates #887. I checked with the same Viewer account and the button is there.
How to set it up
- Ask your agent
“Every night at 2, test signup, checkout and export in a browser on the latest main build. Use Admin, Editor and Viewer accounts. Post what changed since the last good run in #qa.”
What it won't do
It follows the steps you described, so a bug outside those steps goes unseen. It reports the likely cause, but a developer confirms it.
Questions
The ones you describe, such as signup, checkout and export. Add or change flows by telling Maito in Slack.
Yes. Store a login for each role in the Vault and it runs the flow as each, which catches role-only bugs.
It compares the last green run with the failure and checks which merged pull requests landed in between.
