Our first series on engagement discussed activity feed and social product environments, which offer a mix of user-generated and professional content. In this series, we explore engagement on platforms that offer purely professional content. In the first part, we described how production of evergreen professional content makes your product truly engaging. In part two of this series, we focus on how to make content relevant by making the right recommendations.
To recommend well, you must understand the total inventory available to each user, gather implicit and explicit signals about your users, and use those signals to anticipate user behavior (prediction) and determine importance (relevancy) of each inventory to each user. An effective recommendation system must, therefore, include a prediction algorithm that can assign a numerical “relevancy score” to each inventory-user pair.
CONNECTIONS AND INVENTORY
Unlike a social platform such as Facebook, Snapchat or Instagram, which includes user-generated as well as professional content, a platform such as Netflix generally makes its entire inventory available to each subscriber. Thus, inventory is the same for each user, and consumption and engagement depend heavily on how users interact with recommendations, search, browsing, and subscriptions to specific programming or channels.
As a product grows and more content is produced, most users will subscribe to more content and spend more time on the platform — thereby increasing the number of connections. The amount of content consumed will ultimately be a strong indicator of long-term retention and engagement of the user.
Metrics to track
These metrics will help you understand connections on your platform. Consider segmenting them by country, language, type (news, movies, etc.), original vs. licensed, format (text, video, etc.) and/or platform (iOS, Android, desktop).
- Inventory available: The more overall inventory available via recommendations, search and browse, the more likely it is users will find something they want.
- Consumption of available inventory: Know how much a user is consuming relative to their available recommended inventory.
- Number of connections: The more a user is connected to relevant content, the more likely they are to consume content.
A product’s signals comprise all available information on users and their content preferences and can help you predict whether a given user will engage with a given piece of content. Examples of signal categories are given in the table below; note that some categories include hundreds of individual types of signals and that this list is not comprehensive.
PREDICTIONS AND RELEVANCE
Because users’ past behavior is predictive of their future behavior, a machine learning model can use signals like those above to determine to a certain degree of confidence whether a given user will watch a specific content and produce a relevancy score specific to each content-user pair. When each piece of content in your inventory has such scores, your sorting algorithm can then place them in the order they will appear to each user.
These predictions are challenging for multiple reasons. Watching a piece of content for a few minutes does not reveal whether the user liked it; perhaps they did not, perhaps they simply got distracted and forgot to return. Even completing an episode or more does not necessarily indicate they enjoyed it. A high rating, on the other hand, is more instructive. It’s important to take care in determining which signals will inform your relevancy scores, and to what extent. Choosing and properly weighting each function is as much art as science, and can be quite complex: How much should a partial viewing weigh? How important is recency? Clicking on a video?
In addition, the relevancy score for each post-user pair should reflect not only the predictions derived from your signals but your product’s optimization function. Based on your company’s mission, you may decide to optimize for time spent, for example, or for number of sessions or click-through rate. According to Netflix, their business objective is to maximize member satisfaction and month-to-month subscription retention, both of which correlate with maximizing consumption of video content. Therefore, Netflix optimizes its algorithms to give the highest scores to titles a user is most likely to watch and enjoy. Ultimately, a recommendation is effective when it optimizes for your overall goals for the product.
- To recommend well, you must understand the total inventory available to each user, gather implicit and explicit signals about your users, and use those signals to predict user behavior.
Follow us on Medium for weekly updates.
This work is a product of Sequoia Capital’s Data Science team. Chandra Narayanan and Hem Wadhar wrote this post. See the full data science series here. Please email firstname.lastname@example.org with questions, comments and other feedback.