Investigate a drop in a metric
When sign-ups or activation drop, get a first answer on why.
Product · Software · When you ask · Updated
Slack
Amplitude
Mixpanel
PostHog
Datadog
Sentry
New Relic
GrafanaHow do you investigate a drop in a metric?
To investigate a drop in a metric, ask Maito in Slack which metric and which week. It reads Amplitude, Mixpanel or PostHog, splits the drop by step, device and release, checks logs for errors, and replies with where it starts and the likely cause. It also checks the code behind the step.
How it's done today
Analyst queue is two weeks long.
What Maito does
- Pull the metric over time
- Check releases, campaigns and errors around the drop
- Suggest the likely cause
Example
@Maito Why did activation drop last week?
Activation fell from 41% to 33% for the week of Oct 12, while sign-ups held at about 1,150. The drop starts Tuesday Oct 13, the day release 2.31 added the "Connect your data" step to onboarding. Users who skip it activate at 19%. Users who finish it activate at 52%. Skip rate went from 12% to 38%. The catch: 70% of the skips are on mobile web. In onboarding/ConnectData.tsx the step has a fixed height, so on screens under 700px the Continue button sits below the fold. Sentry shows no error spike, so this is a layout problem, not a bug. Numbers are from PostHog; I checked the release log in the repo. Want me to open a branch with a fix for the button? You'd review it before anything ships.
Leo (9:30 AM):
Why did activation drop last week?
Maito (9:36 AM):
Activation fell from 41% to 33% for the week of Oct 12, while sign-ups held at about 1,150.
The drop starts Tuesday Oct 13, the day release 2.31 added the "Connect your data" step to onboarding. Users who skip it activate at 19%. Users who finish it activate at 52%. Skip rate went from 12% to 38%.
The catch: 70% of the skips are on mobile web. In onboarding/ConnectData.tsx the step has a fixed height, so on screens under 700px the Continue button sits below the fold.
Sentry shows no error spike, so this is a layout problem, not a bug. Numbers are from PostHog; I checked the release log in the repo.
Want me to open a branch with a fix for the button? You'd review it before anything ships.
How to set it up
- Set up
Nothing to set up. Ask in Slack or by email whenever you need it.
- Connect analytics
- Amplitude
- Mixpanel
- PostHog
- Connect logs
- Datadog
- Sentry
- New Relic
- Grafana
Not connected yet? Your agent sends you a link to sign in.
What it won't do
Analytics shows what users did, not why. A cause that is not in the data or the code is reported as unknown.
Questions
Amplitude, Mixpanel or PostHog, with Datadog, Sentry, New Relic or Grafana for errors.
It finds where the drop starts and the most likely cause, with the numbers behind it. Your team confirms it.
Yes. Set a Loop and it checks on a schedule and posts only when something moves.
