Monthly Archives: April 2013

Measuring User Engagement

Together with Heather O’Brien and Elad Yom-Tov, we will be giving a tutorial at the International World-Wide Web Conference (WWW), 13-17 May 2013, Rio de Janeiro. Here is a description of our tutorial. We will add slides and a bibliography soon.

Introduction and Motivations
In the online world, user engagement refers to the quality of the user experience that emphasizes the positive aspects of the interaction with a web application and, in particular, the phenomena associated with wanting to use that application longer and frequently. User engagement is a key concept in the design of web applications, motivated by the observation that successful applications are not just used, but are engaged with. Users invest time, attention, and emotion in their use of technology, and it must satisfy both their pragmatic and hedonic needs and expectations. Measurement is key for evaluating the success of information technologies, and is particularly critical to any web applications, from media to e-commerce sites, as it informs our understanding of user needs and expectations, system design and functionality. For instance, news portals have become a very popular destination for web users who read news online. As there is great potential for online news consumption but also serious competition among news portals, online news providers strive to develop effective and efficient strategies to engage users longer in their sites. Measuring how users engage with a news portal can inform the portal if there are areas that need to be enhanced, if current optimization techniques are still effective, if the published material triggers user behavior that causes engagement with the portal, etc. Understanding the above is dependent upon the ability to measure user engagement. The focus of this tutorial is how user engagement is currently being measured and future considerations for its measurement.

User engagement is a multifaceted, complex phenomenon; this gives rise to a number of potential approaches for its measurement, both objective and subjective. Common ways of measuring user engagement include: self-reporting, e.g., questionnaires; observational methods, such as facial expression analysis, speech analysis, desktop actions, etc.; neuro-physiological signal processing methods, e.g., respiratory and cardiovascular accelerations and decelerations, muscle spasms, etc.; and web analytics, online behavior metrics that assess users’ depth of engagement with a site. These methods represent various tradeoffs between the scale of data analyzed and the depth of understanding. For instance, surveys are small-scale but deep, whereas clicks can be collected on a large-scale but provide shallow understanding.

The tutorial will start with a definition of user engagement and discuss the challenges associated with its measurement. The tutorial will then have two main parts. Part I will describe self-report measures, physiological measures, and web analytics. We aim to provide a full understanding of each type of approach, including methodological aspects, concrete findings, and advantages and disadvantages. Part II will concentrate on advanced aspects of user engagement measurement, and is comprised of three sub-sections. We will look at (1) how current metrics may or may not apply to the mobile environment; (2) the relationship between user engagement on-site with other sites in terms of user traffic or stylistics; and finally (3) the integration of various approaches for measuring engagement as a means of providing a deeper and more coherent understanding of engagement success. The tutorial will end with some conclusions, open research problems, and suggestions for future research and development.

Part I – Foundations

Approaches based on Self-Report Measures
Questionnaires are one of the most common ways of gathering information about the user experience. Although self-report measures are subjective in nature, they have several advantages, including being convenient and easy to administer, and capturing users’ perceptions of an experience at a particular point in time. The fundamental problem is that questionnaires are seldom subjected to rigorous evaluation. The User Engagement Scale (UES), a self-report measure developed by O’Brien and colleagues in 2010 will be used to discuss issues of reliability and validity with self-report measures. The UES consists of six underlying dimensions: Aesthetic Appeal, Perceived Usability, Focused Attention, Felt Involvement, Novelty, and Endurability (i.e., users’ overall evaluation). It has been used in online web surveys and user studies to assess engagement with e-commerce, wiki search, multimedia presentations, academic reading environments, and online news. Data analysis has focused on statistically analyzing the reliability and component structure of the UES, and on examining the relationship between the UES and other self-report measures, performance, and physiological measures. These findings will be shared, and the benefits and drawbacks of the UES for measuring engagement will be explored.

Approaches based physiological measures
Physiological data can be captured by a broad range of sensors related to different cognitive states. Examples of sensors are eye trackers (e.g., difficulty, attention, fatigue, mental activity, strong emotion), mouse pressure (stress, certainty of response), biosensors (e.g., temperature for negative affect and relaxation, electrodermal for arousal, blood flow for stress and emotion intensity), oximiters (e.g., pulse), camera (e.g., face tracking for general emotion detection). Such sensors have several advantages over questionnaires or online behaviour, since they are more directly connected to the emotional state of the user, are more objective (measuring involuntary body responses) and they are continuously measured. They are, however, more invasive and, apart from mouse tracking, cannot be used on a large-scale. They can nonetheless be highly indicative of immersive states through their links with attention, affect, the perception of aesthetics and novelty – all of which are important characteristics of user engagement. A particular focus in this tutorial will be the usage of mouse pressure, so-called mouse tracking, because of its potential for large-scale measurement. The use of eye-tracking to measure engagement will also be discussed, because of its relationship to mouse movement.

Approaches based on web analytic
The most common way that engagement is measured, especially in production websites, is through various proxy measures of user engagement. Standard metrics include the number of page views, number of unique users, dwell time, bounce rate, and click-through rate. In addition, with the explosion of user-generated content, the number of comments and social network “like” buttons are also becoming widely used measures of web service performance. In this part we will review these measures, and discuss what they measure vis-à-vis user engagement, and consequently their advantages and drawbacks. We will provide extensive details on the appropriateness of these metrics to various websites. Finally, we will discuss recent work on combining these measures to form single measures of user engagement.

Part II – Advanced Aspects

Measuring User Engagement in Mobile Information Searching
Mobile use is a growing area of interest with respect to user engagement. Mobile devices are utilized in dynamic and shifting contexts that form the fabric of everyday life, their portability and functionality make them more suited to some tasks than others, and they are often used in the presence of other people. All of these considerations – context, task, and social situatedness – have implications for user engagement. The Engagement Lab at the University of British Columbia is exploring user engagement with mobile devices in a series of studies. In this section of the tutorial, we will explore the ways in which mobile engagement may differ from engagement with other devices and what the implications of this are for measurement. We will describe both lab and field-based work that we are undertaking, and the measures that we are selecting to capture mobile engagement.

Networked User Engagement
Nowadays, many providers operate multiple content sites, which are very different from each other. For example, Yahoo! operates sites on finance, sports, celebrities, and shopping. Due to the varied content served by these sites, it is difficult to treat them as a single entity. For this reason, they are usually studied and optimized separately. However, user engagement should be examined not only within individual sites, but also across sites, that is the entire content provider network. Such engagement was recently defined by Lalmas et al. as “Networked User Engagement”. In this part of the tutorial we will present recent findings on the optimization of networked user engagement. We will demonstrate the effect of the network on user engagement, and show how changes in elements of websites can increase networked user engagement.

Combining different approaches
Little work has been done to integrate these various measures. It is important to combine insights from big data with deep analysis of human behavior in the lab, or through crowd-sourcing experiments, to obtain a coherent understanding of engagement success. However, a number of initiatives aiming to combine techniques from web analytics, existing works on user engagement coming from the domains of information science, multimodal human computer interaction and cognitive psychology, are emerging. We will discuss work emerging in these directions, and in particular studies related to mapping mouse tracking and qualitative measurement of user engagement, and the challenges in designing experiments, and interpreting and generalizing results.

Your news audience in Twitter – Discover your curators

@SNOW/WWW, 2013, by Janette Lehmann, Carlos Castillo, Mounia Lalmas, and Ethan Zuckerman

Original post at WWW Workshop on Social News on the Web

flickr - striatic - 436654901_e1b0204d14_o_edited

Information between journalists and their audience in social media flows in both ways. A recent study from the Oriella PR Network showed that over 54% of journalists use online social media platforms (Twitter, Facebook, and others) and 44% use blogs to find new story angles or verify the stories they work on. There are now platforms, such as Storyful, that provide user lists of high quality, developed by journalists for journalists.

Our starting point is the community of engaged readers of a news story — those who share a particular news article through Twitter. We refer to them as a transient news crowd, in analogy with the group of passers-by that gathers around unexpected events such as accidents in a busy street. The question is whether the users of such a crowd can provide further valuable information related to the story of the news article.

Many members of news crowds are far from being passive in their consumption of news content. They are news curators, because they filter, select, and disseminate carefully selected news stories about a topic.

A famous example for this type of news curator is Andy Carvin (@acarvin), who mostly collects news related to the Arabic world. He became famous for his curatorial work during the Arab Spring, where he aggregated reports in real time and tweeted up to thousands of tweets per day. We expect that among the users who share an article in Twitter are also other curators like Andy Carvin who may follow-up with further tweets.

We have observed that basically all news stories have a set of Twitter users who may be potential news curators for the topics of the story. For instance, among the people who tweeted the Al Jazeera’s article “Syria allows UN to step up food aid” (posted January 2013), there are at least two news curators: @RevolutionSyria and @KenanFreeSyria.

However, not everybody can be considered a news curator. Some people tweet one piece of news that was interesting to them and move on. Others tweet a wide range of news stories. Curators are individuals who carefully follow a story or related set of stories. In our SNOW 2013 work “Finding News Curators in Twitter”, we defined a set of features for each user and demonstrated that they can be used to automatically find relevant curators among the audience. The features describe the visibility, tweeting activity and the topical focus of a user. We collected news articles published in early 2013 of BBC World Service (BBC) and Al Jazeera English (AJE). Then, we followed the users that posted a specific article and analyzed their tweeting behavior. Our results reveal that 13% of the users from AJE and 1.8% of the users from BBC world are possible news curators.

The roles of curators in a crowd…

Some news curators are more focused than others. For instance, @KeriJSmith, a self-defined “internet marketer” tweets about various interesting news on a large variety of topics, while others are more selective. A famous example is Chan’ad Bahraini (@chanadbh) who tweets about Bahrain. Whether a user is topic-focused or not can be determined, for instance, by the number of different sections of a news web site s/he is tweeting about. If these sections differ (e.g. from finance to celebrities), we can assume that the user is less focused.

Considering only the topical focus of a user is not sufficient when identifying story curators. A significant amount of Twitter accounts operate as news aggregators – collecting news articles automatically (e.g. from RSS feeds) and posting their corresponding headlines and URLs to Twitter (45% in Al Jazeera English, 65% in BBC world). They can be identified easily, as most or all of their tweets contain URLs and they do not tend to interact much via messages with other users.

The majority of news aggregators post many tweets per day related to breaking news and top stories, e.g. @BreakingNews. Only a minority is focused on more specific topics, and thus constitutes topic-focused aggregators. The user @RevolutionSyria, for instance, distributes automatically news articles about the civil war in Syria. Whether the automatic generated tweets provide interesting content to a topic is questionable. Nonetheless, some news aggregators seem to be considered valuable by users, as in the case of @RevolutionSyria who has around 100,000 followers at the time of this writing.

In short, our current research deals with identifying crowds, curators, and aggregators. For more details you can check our articles and presentations:

  • Janette Lehmann, Carlos Castillo, Mounia Lalmas and Ethan Zuckerman: Finding News Curators in Twitter. To be presented at the WWW Workshop on Social News On the Web (SNOW), Rio de Janeiro, Brazil.
  • Janette Lehmann, Carlos Castillo, Mounia Lalmas and Ethan Zuckerman: Transient News Crowds in Social Media. To be presented at the Seventh International AAAI Conference on Weblogs and Social Media (ICWSM), 8-10 July 2013, Cambridge, Massachusetts.

Photo credits: Hobvias Sudoneighm (Creative Commons BY).