From the Lab to the Market

What makes an ad preferred by users?

This is the first blog post on a paper that will be presented at WWW 2016 [1], on our work on advertising quality. The focus of this work was the pre-click quality of native advertisements. This work is in collaboration with Ke (Adam) Zhou, Miriam Redi and Andy Haines.

In online services, native advertising has become a very popular form of online advertising, where the ads served reproduce the look-and-feel of the platform in which they appear. Examples of native ads include suggested posts on Facebook, promoted tweets on Twitter, or sponsored contents on Yahoo news stream. On the right, we show an example of a native ad (the second item with the “dollar” sign) in a news stream on a mobile device.

Promoting relevant and quality ads to users is crucial to maximize long-term user engagement with the platform. In particular, low-quality advertising has been shown to have detrimental effect on long-term user engagement. Low quality advertising can have even more severe consequences in the context of native advertising, since native advertisement forms an integrated part of the user experience of the product. For example, a bad post-click quality (quantified by short dwell time on the ad landing page) in native ads can result in weaker long-term engagement (e.g. fewer clicks).

Here we focus on the pre-click experience, which is concerned with the user experience induced by the ad creative before the user decides (or not) to click.

The ad creative is the ad impression shown within the stream, and includes text, visuals, and layout. Due to the low variability in terms of ad formats in native advertising, the content and the presentation of the ad creative are extremely important to determine the quality of the ad.

Our first step was to understand ad quality from a user perspective, and infer the underlying criteria that users assess when choosing between ads.

To this end, we designed a crowd-sourcing study to spot what drives users’ quality preferences in the native advertising domain.

We extracted a sample of ads impressed on Yahoo mobile news stream.  To ensure diversity and the representativeness of our data in terms of subjects and quality ranges, we uniformly sampled a subset of those ads from (1) different click-through rate quantiles;  and (2) five different popular topical categories: “travel”, “automotive”, “personal finance”, “education” and “beauty and personal care”.

We used Amazon Mechanical Turk to conduct our study. We showed users pairs of native ads, and asked them to indicate which ad they prefer, and the underlying reasons for their choice. To eliminate the effect of ad relevance, we presented the users with topically-coherent ads (e.g., ads from the same subject category, such as “beauty”), assuming that, for example, when users are comparing two beauty ads, the preference depends mostly on the ad quality.

Once users chose their preferred ad, we asked them the reasons why they chose the selected ad. To define such options, we resorted to existing user experience/perception research literature. We were inspired by the UES (User Engagement Scale) framework, an evaluation scale for user experience capturing a wide range of hedonic and cognitive aspects of perception, such as aesthetic appeal, novelty, involvement, focused attention, perceived usability, and endurability. Moreover, previous studies in the context of native advertising investigated user perceptions of native ads with dimensions such as “annoying”, “design”, “trust” and “familiar brand”. Similarly, researchers have studied the amount of ad “annoyingness” in the context of display advertising, showing that users tend to relate ad annoyance with factors such as advertiser reputation, ad aesthetic composition and ad logic. Based on these, we provided users with the following options as underlying reasons of their choice:

• the brand displayed
• the product/service offered
• the trustworthiness
• the clarity of the description
• the layout
• the aesthetic appeal

Users were asked to rate each on a five-grade scale: 1 (strongly disagree), to 5 (strongly agree) or NA (not available).

We report  in the following table the percentage of judgements that, for each factor, is assigned to grades 4 or 5 (the user highly agrees this factor affects his or her ad preference choice).

Underlying reasons of users’ preference of ad pairs based on ad creative.

The most important factors are, in order of importance:

Aesthetic appeal > Product, Brand, Trustworthiness > Clarity > Layout

where “>” represents a significant increase (in ad preferences). Further test showed that, apart from the brand factor, there were not any significant differences. This suggests that the factors affecting user preferences generalize across ad categories.

However, for different ad categories, compared to the general pattern, we observe few small differences. Aesthetic appeal is more important for Automotive, Beauty and Education, than Personal Finance and Travel. For the Travel category, where most ad images were indeed beautiful, aesthetics did not affect much compared to others. For Beauty and Education categories, the product advertised was the most important factor (other than aesthetic appeal) affecting user choices; for Automotive, the brand was crucial. For Personal Finance category, the clarity of the description had a big impact on the user perception of the quality if the ad.

This study provided us important insights into how users perceive the quality of native ads. In a future blog post, I will discuss how we map these insights to engineered features, which we then use to predict the pre-click experience.

• [1] K. Zhou, M. Redi, A. Haines and M. Lalmas. Predicting Pre-click Quality for Native Advertisements, 25th International World Wide Web Conference (WWW 2016), Montreal, Canada, April 11 to 15, 2016.