In this final post of our seven-part series on analyzing metric changes, we offer additional best practices for addressing metric changes due to the factors discussed in previous posts: product changes, seasonality and other behavioral changes, mix shift and data quality.
Once you have confirmed that there is indeed a change in metric worth investigating, you need to develop a systematic and structured approach to identifying and attempting to eliminate each possible cause.
The first step is selecting two points in time that best represent the change in metric you are investigating. (As explained in Part 4, the larger the change and shorter the time frame, the easier it will be to identify the root cause.) Next, you should ask lots of questions about what could have caused the change in your key metric. Once you have a comprehensive list of hypotheses, eliminate or investigate factors one by one:
DATA QUALITY
Investigate issues with data quality first, as they may be the easiest to identify. Look for logging issues related to product changes — for example, a bug that incorrectly records DAU (daily active users) for a certain locale, language, country, device, etc.
- To localize such an issue, investigate whether the change is systemic across all dimensions or specific to some dimensions.
- Examine other correlated metrics for similar changes. For example, if number of sessions is correlated with DAU, and you see a change in DAU but not number of sessions, a logging bug may be the problem.
Refer to Part 6 for details on data quality issues.
PRODUCT CHANGES
List changes made to the product in the given timeframe. (If no changes were made, you can eliminate this factor, but note it’s possible to forget. Find a way to account for this by tracking changes.) If you have an experimentation framework (A/B testing), quantify the impact each product change would have on your key metric. Look for behavioral changes due to product changes. Examine behavioral changes by group (country, device, etc.) to determine whether the changes are localized, then examine the time at which you saw the metric change. If the metric change happened outside of the time period you would expect based on the timing of the product change, it is unlikely the latter caused the former. Remember, too, that network effects can sometimes carry the impact of an issue beyond the population that is primarily affected; for example, if a bug prevents people in Israel from using a communication platform, this could lower engagement of people in other countries, as well.
BEHAVIORAL CHANGES
Seasonality is the generally the largest contributor to behavioral change, though external events and competition may also influence this factor. For specific guidance on investigating behavioral changes, see Part 3 and Part 4.
MIX SHIFT
As explained in Part 5, the first step to diagnosing mix shift is hypothesizing the dimension in which you expect it to occur. Part 5 offers specific guidance on quantifying the effect of mix shift. Note mix shift may be a strong factor in long-term changes, but likely will not be the source of changes week over week.
TAKEAWAYS
- Shifts in metrics are almost always due to data quality, product changes, behavioral changes and/or mix shift.
- Because shifts are rarely due to just one factor, they can be difficult to diagnose.
Follow us on Medium for weekly updates.
This work is a product of Sequoia Capital’s Data Science team. Jamie Cuffe, Avanika Narayan, Chandra Narayanan, Hem Wadhar and Jenny Wang contributed to this post. Please email data-science@sequoiacap.com with questions, comments and other feedback.
