From the Lab to the Market

Bounce, Shallow, Deep & Complete: Four Levels of User Engagement in News Reading

This is the second blog post on a paper that will be presented at WSDM 2016 [1], on metrics of user engagement using viewport time. This work is in collaboration with Dmitry Lagun, and was carried out while Dmitry was a student at Emory University, and as part of a Yahoo Faculty Research and Engagement Program.

In a previous blog post, I motivated using viewport time (the time a user spends viewing an article at a given viewport position) as a coarse, but more robust instrument to measure user attention during news reading. In this blog, I describe how to use viewport time to classify each individual page view into one of the following four levels of user engagement:

Four levels of User Attention

• A Bounce indicates that users do not engage with the article and leave the page relatively quickly. We adopt 10 seconds dwell time threshold to determine a Bounce. Other thresholds can be used, for example accounting for genre (politics versus sport) or device (mobile versus tablet)
• If the user decides to stay and read the article but reads less than 50% of it, we categorize such a page view as Shallow engagement, since the user has not fully consumed the content. The percentage of article read is defined as the proportion of the article body (main article text) having a viewport time longer than 5 seconds. Note that using 50% is rather arbitrary and used only to distinguish between extreme cases of shallow reading and consumption of the entire article.
• On the other hand, if the user decides to read more than 50% of the article content, we refer to this as Deep engagement, since the user most likely needs to scroll down the article, indicating greater interest in the article content.
• Finally, if after reading most of the article the user decides to interact (post or reply) with comments, we call such experience Complete. The users are fully engaged with the article content to the point of interacting with its associated comments.

To understand what insights these engagement levels actually bring, we compare them with four sets of measures, by reporting the mean and standard errors between them and viewport time broken down for the engagement levels. This analysis is based on the viewport data for 267,210 page views on an online news website from Yahoo.

• Dwell time.
• Viewport time for article header (usually a title which may include small image thumbnail), body (main body of the article), and comment block.
• Percentage of the total viewport time spent viewing one of the above mentioned regions.
• Comment clicks showing the number of clicks on the comment block.

Means and standard errors of the fined grained measures for Bounce, Shallow, Deep and Complete (all with statistical significant differences)

Dwell time and viewport time on head, body and comment increase from Bounce to Complete. We also note that the distribution of the percentage of time among these blocks changes in an interesting manner. The viewport time on head steadily decreases from 0.31 for Bounce to 0.09 for Complete indicating that users spend an increasing amount of time reading content deeper in the article. The percentage of article read steadily increases from Bounce to Complete, as expected. With respect to this, Bounce (12%) and Shallow (23%) clearly represent low levels of engagement with the article, since less than 25% of the article was read. On the other hand, Deep and Complete correspond to the situations when the majority (83%) of the article was read. The number of comment clicks is highest for Complete (3.14), followed by Shallow (0.43), suggesting that users may engage with comments even if they do not read a large proportion of the article.

Mean viewport time at different viewport positions for each of the levels of user engagement (thickness of the line corresponds to the standard error of the mean)

Finally, we compute the average viewport time at varying vertical position. Each of the four curves corresponds to one of the engagement levels. For the page views in the Bounce case, users rarely scroll down the page, whereas many users in the Shallow case spend approximately another 5 seconds of viewport time at lower scroll positions. Deep engagement is characterized by significant time spent on the entire article (peak at the first screen amounts to about 50 seconds) with a steady position decay of the viewport time towards the bottom. Interestingly, the viewport time profile for Complete engagement no longer monotonically decays with the position; instead it has a bi-modal shape. This could be due to a significant time users spend viewing and interacting with comments, which are normally placed right after the main article content.

Our analysis shows that the proposed four user engagement levels are intuitive, and bring more refined insights about how user engage with articles, than using dwell time alone. We recall that the engagement levels can be derived using the viewport time information, which can be computed through scalable and non-intrusive instrumentation. In a future blog post, I will describe how viewport time can be used to predict these levels of engagement based on the textual topics of a news article.

• [1] D. Lagun and M. Lalmas. Understanding and Measuring User Engagement and Attention in Online News Reading, International Conference On Web Search And Data Mining (WSDM 2016), San Francisco, USA, 22-24 Fabruary 2016.