Bulletin, December/January 2006

Sponsored Search: A Brief History

by Daniel C. Fain and Jan O. Pedersen

Daniel C. Fain is a director of research at Yahoo! Inc. Jan O. Pedersen is chief scientist for the search and marketplace division at Yahoo! Inc. They can be reached at 701 First Ave., Sunnyvale CA 94089-0703; faind at yahoo-inc.com and jpederse at yahoo-inc.com, respectively. 

Web search is a fundamental technology for navigating the Internet. It has become an essential part of the online experience – 98.8% of Internet users utilize search (iProspect, 2004) - yet, it is provided free to consumers. Sponsored search, the delivery of relevance - targeted text advertisements as part of the search experience, makes this possible. In the week ending August 21, 2005, search engines displayed 13 billion sponsored results (Nielsen//NetRatings, 2005). Sponsored search has evolved to satisfy users' need for relevant search results and advertisers' desire for qualified traffic to their websites, and it is now considered to be among the most effective marketing vehicles available.

The basic elements of sponsored search are

  • Advertiser-provided content: a set of advertiser hyperlinks annotated with keyword tags, titles and descriptions.
  • Advertiser-provided bids that value traffic on specified concepts or keywords.
  • Review process combining manual and automated methods to ensure that advertiser content is in fact relevant to the target keyword
  • Matching of advertiser content to user queries as they are received by a search engine
  • Display of advertiser content in some rank order in some placement alongside other algorithmic (that is., non-sponsored) search engine content
  • Processes that gather data, meter clicks and charge advertisers based on consumer clicks on their displayed content

In 1998, GoTo, later Overture Services, was the first to combine these elements. Yahoo! acquired Overture in 2003, re-branding it to Yahoo! Search Marketing in 2005. BeFirst - now MIVA - followed with a similar product in 1999. Google adopted the model and modified it to incorporate click feedback in 2002. In 2005, Ask Jeeves adopted it, and MSN Search extended it to support behavioral targeting. Sponsored search engines display results not only on their own sites, but also in space rented on other sites. For example, when a user searches at MSN Search, the search query is passed to Yahoo!’s sponsored search engine, which returns results to the MSN server, which in turn renders the page that the end user sees. Similarly, Google rents space on AOL.

In the early days of Web search, sites used annotation via <META> tags that contained keywords supposedly describing the site's content. Some sites used these tags to outrank other sites that may have been more popular or relevant. Even worse, some sites - in particular, pornographic ones - manipulated <META> tags and appeared in the results for essentially irrelevant queries. To address this problem, GoTo focused on roughly the top 1,000 queries, employing editors to create handcrafted result pages with images, product and service offers and informational links. When GoTo introduced sponsored search, each listing was associated with a keyword. The keyword set was open ended, and advertisers could request to add new ones. 

In other words, GoTo's editors ensured the relevance of the advertiser content to the keyword. Given the rapid growth of the sponsored search industry, this process is now partially automated; t. The majority of Yahoo!'s sponsored search listings are initially evaluated by software.

In the beginning, Internet advertising mimicked offline advertising. Viewers of print ads are more likely to remember ones that are more graphical and less text-oriented (Du Plessis, 2005, p. 128). HotWired introduced graphical banner advertisements to the Web in 1994. Search engines relied on banner advertising before the adoption of sponsored search, so they faced a dilemma - : keep users on the site as long as possible to view more banners or send the users promptly to the sites appearing in the search results. 

Paid search reconciled this dilemma by tying the search engine’s revenue to the act of transferring the user to an advertiser's site. In 1996, the search engine Open Text briefly offered preferred listings, in which sites would pay to be inserted into the search result set for particular keywords.

From the start, advertisers were concerned with how to measure effectiveness, and publishers were concerned with how to set prices. Some publishers charged a fixed fee to appear on a site, while others such as Netscape and Infoseek in 2005 used the cost per mille (CPM) - the cost to display an advertisement a thousand times - just as for offline advertising. A very basic question was whether the banner advertisements were actually causing users to visit the site. Some advertisers were practicing direct response marketing, which seeks a measurable and attributable response. The easiest response to measure was the click on a hyperlink. In 1996 Yahoo! agreed to charge Procter & Gamble only when users clicked from Yahoo! to a Procter & Gamble Web page. The two companies used the cost per click (CPC) as a common currency, translating the advertiser goal into a basis for pricing. For search advertising, LinkStar introduced CPC pricing in 1996.

Even without a common currency, tools became available that converted between CPM and an advertiser's expected/effective cost per click (eCPC). A natural extension of CPC account management or pricing is cost per action (CPA). Example actions are purchasing or signing up for a new account. By 1997, DoubleClick was already offering CPA pricing. Advertisers could configure their websites to relate a particular action to a unique search. For example, a travel site may know that a passenger who bought an airline ticket arrived at the site by searching on "travel" on a particular sponsored search engine. In 2003, Overture, Google and FindWhat introduced automated tools to enable sponsored search advertisers to measure CPA and set appropriate CPC bids. SNAP adopted CPA as the standard pricing model for its search engine in 2004. It is even possible to determine whether telephone calls were prompted by a Web search by using specific telephone numbers. Ingenio started providing this service in 1999, and both MIVA (previously FindWhat) and MSN integrated it as a pricing option for sponsored search in 2005.

It may be easier for an advertiser to express a goal as CPA, CPC or CPM, depending on the circumstances. The ratio between CPC and CPA is known as the conversion rate. Click-through rate (CTR) is the ratio between cost per impression and CPC. An impression is one display of an advertisement, so the cost per impression is CPM/1000. If the goal is brand awareness, it is sometimes sufficient to see an advertisement without clicking, so there is some dispute over the importance of CTR (Briggs, 1997; Hoffman, 2000).

Both conversion rate and CTR are conditional probabilities: of an action given a click and of a click given an impression. In principle, with enough data, one can directly estimate these probabilities. A sponsored search campaign will usually encompass a huge number of unique queries, and for most of those queries, there will be insufficient data. Somehow, data must be combined across contexts - for example, by grouping queries together or comparing the current campaign to others. Yet, additional variables such as the time of day, or day of week, can be introduced that predict variations in conversion rate and CTR. This variation is one of the reasons that sponsored search bids fluctuate throughout the day. 

In 1997, FlyCast and Narrowline introduced marketplaces that enabled advertisers to manage campaigns across multiple publisher sites, each of which placed a bid for advertisements. In 2003, Yahoo! was the first major sponsored search provider to offer a tool that enabled campaign management across multiple online advertising types, including banners, sponsored search and email campaigns. Both Yahoo! and Google now run continuous auctions, in which a competitor willing to pay more can displace an advertiser at any time. Yahoo! sorts search results first by the preciseness of the match, then by the CPC bid selected by the advertiser. Direct Hit ranked search results using CTR. When Google re-launched AdWords in 2002, they modified Overture’s ranking rules to use CTR*CPC instead of just CPC. Google generalized the CTR prediction, incorporating an analysis of the listing text (Google, 2005). Similarly, Yahoo! calculates a Click Index score by comparing a listing’s observed CTR to a standard. This score is displayed in the advertiser’s interface, but not used when ranking search results. When a listing's score is very low, it is flagged for possible removal.

Yahoo! continues to transform the original sponsored search engine. Increasingly, there is academic interest in sponsored search. Weber and Zheng (2004) proposed ranking search results by combining merchant ratings and bids. Varian (1999) presented an argument for sometimes placing a more obscure interpretation of a query above a more likely interpretation based on the average user’s search cost in the economic sense. Yahoo! helped organize a workshop at the 2005 E-Commerce Conference (http://research.yahoo.com/workshops/ssa2005/sched.html) to bring together the fields of user experience, economics and information retrieval. Sponsored search continues to improve advertisers’ effectiveness in targeting their campaigns and to provide highly relevant results for searches with commercial intent.

For Further Reading

Briggs, R & Hollis, N. (1997). Advertising on the Web: Is there response before click-through? Journal of Advertising Research, 37(2), 33 et seq.

Du Plessis, E. (2005). The Advertised Mind. Sterling, VA: Kogan Page.

Google. (2005). “How are ads ranked?” Available October 14, 2005, at https://adwords.google.com/support/bin/answer.py?answer=6111&topic=115.

Hoffman, D and Novak, T. (2000). How to acquire customers on the Web. Harvard Business Review, 78(3), 179-188.

iProspect. (2004). Search engine user attitudes survey April-May 2004. Available October 14, 2005 at www.iprospect.com/premiumPDFs/iProspectSurveyComplete.pdf 

Nielsen//NetRatings. (2005, September 7). "Sponsored search link placements led online by EBay, Shopping.com Inc., and InterActiveCorp" , according to Neilsen/NetRatings. [(Ppress release]). Nielsen//NetRatings. Available October 14, 2005, at http://direct.www.netratings.com/pr/pr_050907.pdf

Varian, H. (1999). "Economics and search." SIGIR Forum, 33, (1), 1-5. Available October 14, 2005, at www.sims.berkeley.edu/~hal/papers.html.

Weber, T.A., & Zheng, Z. (2004). A model of search intermediaries and paid referrals. Current version of a paper presented at the Workshop on Information Systems and Economics, Barcelona Spain, December 14-15, 2002. (Wharton School OPIM Opinion Paper No. 02-12-01.) Paper available October 14, 2005, through the Social Science Research Network at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=601903