Analyzing Metric Changes Part III: Seasonal Factors

Data Science Team

In Part 2 of this series, we examined the ways product factors can affect metrics. This post will explore seasonal factors, which often cause of dramatic fluctuations in key metrics. People behave differently based on time and circumstance, and seasonality is one of the most prominent influences that can be analyzed using data.

SEASONALITY

Seasonality is a broad label for periodic changes in user behavior. For example, people change their behaviors at the beginning of each year—at least for a short period of time—as people take on New Year’s resolutions. In Figure 1, we see a significant increase after January 1 in the percentage of people exercising 30 minutes or more. Other examples of seasonality include behavioral changes based on time of day, day of the week, season of the year, etc. It is important to separate these from external factors such as the launch of Pokémon GO! which had a similar effect on physical activity (see Figure 1). We’ll discuss these external shifts in next week’s post.

Fig 1

ANALYZING SEASONALITY

Seasonality is often the root cause of changes in key metrics; as such, it is the first behavioral change factor to explore.

To properly analyze any behavioral change, you must understand your product’s overall ecosystem. For example, it might be important to know what young people do during the summer, how middle-aged women shop, how people using Android behave compared to those using iOS or who is likely to be an early adopter of your product.

Understanding this ecosystem allows you to ask good questions and develop good hypotheses, which is how effective analysis of behavioral change almost always begins. Ask yourself: What is the likely root cause of the change in key metric? You might then hypothesize that the reason is seasonality, that users are getting bored or that there has been a change in perception—positive or negative—of the product. Once you have a comprehensive list of hypotheses for why the key metric might have changed, you’re ready to investigate further.

If you are testing a hypothesis that seasonality is a factor in the change, you may take several approaches:

Week-over-week movement Most products exhibit a very strong day-of-the-week effect, because people behave differently on weekdays than they do on weekends. To remove this effect, use seven-day (or rolling seven-day) averaging of the metric of interest. For example, Figure 2 shows strong weekly patterns in usage of Yelp, with increases on weekends. Without 7-day averaging, any week-to-week changes would be lost in day-to-day fluctuations.

Fig 2

(Note social apps often see similar intraday effects, because most people spend far more time on social apps in the evenings than they do during the day. If the lowest level of granularity you studied for such an app was daily use, you would miss that signal completely.)

Day-over-day movement If you are interested in examining day-to-day changes, try overlaying each day of the week as seen in Figure 3. This approach will give you more granular information; for example, this shows us that week 3 has higher highs and lower lows compared to other weeks.

Fig 3

Year-over-year changes Observing annual trends in your data is one of the most powerful ways to identify the effects of seasonality. This method will illuminate the effects of holidays, the back-to-school season, festivals, cold weather, etc. and help you determine whether those impacts on your key metrics are amplifying or diminishing over time.

For example, the chart on the left in Figure 4 shows a seven-day rolling average of revenue for an e-commerce company that is clearly rising. In some cases, where changes are less obvious, a longer rolling average such as 28 days (see the chart on the right in Figure 4) can help demonstrate how a metric is trending over time.

Fig 4

28-day rolling averages, however, tell us relatively little about how a key metric is increasing annually; we see a strong increase toward the end of the year (for the holiday season), but few other patterns are discernible. Instead, we should look at the year-over-year metric (see Figure 5, left), which will tell us much more about seasonal patterns.

The chart on the right in Figure 5 shows the same 28-day rolling average pattern year-over-year; the changes within the year are much more apparent. We can now clearly see that revenue between May and July is influenced by seasonality. Revenue decreases throughout the month of May, begins to sharply increase in early June, then decreases again until around July 4, at which point we see a large bump in both 2016 and 2017, likely due to a sale event. Understanding that these events happen every year will help you plan for the future.

Fig 5

Year-over-year changes can be even better understood by plotting the ratio of revenue from one year to the next (see Figure 6, left). Revenue growth declined year-over-year throughout 2016, but in 2017, things started to improve significantly. Indexing to 1 (see Figure 6, right) is another powerful method to see how things change from the beginning of a year. In this case, it shows 2017 was the best year from a year-over-year perspective, as the company managed to turn things around.

Fig 6

Comparison You can also analyze seasonality by comparing your key metrics to other apps with similar demographics. For example, if your app targets younger populations and your key metrics shift in August and September, try to obtain data from another app with a similar demographic, such as Snapchat, to see if it experienced a similar shift. If so, the back-to-school effect likely caused your shift.

In this analysis, it is best to average over at least seven days and index to 1, in order to identify macro trends. Comparing your metric to those of apps in different locations (and with different school calendars) can also help identify this effect.

TAKEAWAYS

Follow us on Medium for weekly updates.


This work is a product of Sequoia Capital's Data Science team. Jamie Cuffe, Avanika Narayan, Chandra Narayanan, Hem Wadhar and Jenny Wang contributed to this post. Please email data-science@sequoiacap.com with questions, comments and other feedback.