Investigate an error spike
When errors jump, get a first look at the cause before anyone opens a laptop.
Engineering · Software · When something happens · Updated
Slack
Datadog
Sentry
New Relic
GrafanaHow do you investigate an error spike?
To investigate an error spike, run a Loop in Maito that starts from your monitoring alert. It compares the error rate with normal, reads the logs and error tracker in Datadog, Sentry, New Relic or Grafana, links the spike to the recent deploy, and posts the likely cause with options in your on-call channel.
How it's done today
On-call engineer starts from scratch at 2am.
What Maito does
- Read the alert
- Check recent deploys, logs and errors
- Suggest the likely cause and fix
Example
Loop run: Datadog alert · Checkout 5xx above 5% for 5 min
Checkout errors are at 9.4% (normal is 0.3%). They began at 2:41 AM, 10 minutes after deploy 4.12 went out. Likely cause: Sentry shows 1,284 events of one error, TypeError: cannot read 'country' of null, in tax/calculateTax.ts line 77. That code came in with PR #868. Catch: it only hits guest checkouts, which have no customer record. Logged-in checkouts are fine, which is why it passed testing. Options: • Roll back to 4.11 • Patch: a 3-line null check, ready on branch fix/tax-null-customer, tests passing I haven't rolled back or deployed anything. Alex, your call.
@Maito rolling back now. Open the patch as a PR for the morning.
PR #911 is open with a guest-checkout test. It's not merged.
Maito (2:52 AM):
Checkout errors are at 9.4% (normal is 0.3%). They began at 2:41 AM, 10 minutes after deploy 4.12 went out.
Likely cause: Sentry shows 1,284 events of one error, TypeError: cannot read 'country' of null, in tax/calculateTax.ts line 77. That code came in with PR #868.
Catch: it only hits guest checkouts, which have no customer record. Logged-in checkouts are fine, which is why it passed testing.
Options:
• Roll back to 4.11
• Patch: a 3-line null check, ready on branch fix/tax-null-customer, tests passing
I haven't rolled back or deployed anything. Alex, your call.
Alex (2:58 AM):
rolling back now. Open the patch as a PR for the morning.
Maito (3:01 AM):
PR #911 is open with a guest-checkout test. It's not merged.
How to set it up
- Ask your agent
“When Datadog alerts that checkout errors are above 5% for 5 minutes, look into it and post what you find in #oncall. Check recent deploys, Sentry and the code. Don't roll anything back.”
- Connect logs
- Datadog
- Sentry
- New Relic
- Grafana
- Connect error tracker
- Sentry
- Datadog
Not connected yet? Your agent sends you a link to sign in.
What it won't do
It reads what your monitoring tools record. A cause that leaves no trace in logs or errors will be reported as unknown, not guessed.
Questions
An alert from your monitoring tool through a webhook, or a mention in Slack. You can also have it check every few minutes.
No. It lists options like a rollback or a patch and waits for a person to choose.
It says so, lists what it checked and what it ruled out, and gives the on-call engineer a shorter starting point.
