A survey on click modeling in web search

Size: px
Start display at page:

Download "A survey on click modeling in web search"

Transcription

1 A survey on click modeling in web search Lianghao Li Hong Kong University of Science and Technology

2 Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

3 Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

4

5 Search engine marketing

6 Generalized second-price auction

7 Search advertising demo

8 Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

9 Why do we need click prediction? Revenue is highly influenced by click probability prediction. Search engines rank ads with expected revenue E[revenue] = P ad (click) GSP(ad)

10 How to predict click behavior? Click-through logs help! Figure: Ranking presented for the query support vector machine

11 How to predict click behavior? To predict clicks by counting! P ad (click) = # of clicks # of impressions However, that is far from satisfaction clicks are biased due to the user browsing behavior long tail and cold start problems

12

13 How to predict click behavior? Long tail and cold start problems

14 How to predict click behavior? Long tail and cold start problems

15 Long tail query demo: Google vs. Bing

16 A unified framework for click modeling Problem definition Definition 1: (Click modeling) Let random variable u denotes a user, q denotes a query issued by the user, a denotes an ad, r is the position of the ad. The binary variable c is 1 if the ad is clicked and 0 otherwise. Let L denotes the impression list and S denote the click sequence. Click modeling aims to explain observed click events. The shorthand is: P(c, q, a, u, r, L, S) Goals of click modeling 1 To estimate the actual ad relevance from biased click-through logs 2 To predict P(c = 1 q, a, u, r, L, S) for future impressions

17 An overview of click models Hypotheses in click modeling To model click events, we have to incorporate proper browsing hypotheses (i.e., generative process). The main hypotheses include: Unbiased hypothesis: P(c q, a, u, r, L, S) = P(c q, a) Position bias hypothesis: P(c q, a, u, r, L, S) = P(c q, a, r) Depend on click pattern: P(c q, a, u, r, L, S) = P(c q, a, r, S) : P(c q, a, u, r, L, S) = P(c q, a, r, L) Depend on user intent: P(c q, a, u, r, L, S) = P(c q, a, u, r)

18 Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

19 Unbiased hypothesis Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

20 Unbiased hypothesis Unbiased hypothesis: Basic hypothesis Basic hypothesis In the basic hypothesis, there is no bias associated with the observed clicks. This leads to the simplest model: P(c q, a, u, r, L, S) = P(c q, a) Remark In the basic hypothesis, the click probability is dominated by the relevance between query q and ad a.

21 Position bias hypothesis Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

22 Position bias hypothesis Position bias hypothesis: Examination hypothesis Examination hypothesis (WWW 07, Richardson et al.) Examination hypothesis assumes that an ad be clicked must be both examined (i.e. e = 1) and relevant: P(c = 1 q, a, u, r, L, S) =P(c = 1 q, a, r) Independence assumption = P(c = 1 e, q, a, r)p(e q, a, r) e {0,1} =P(c = 1 e = 1, q, a)p(e = 1 r) Examination hypothesis Novelty: The first attempt to model position bias

23 Position bias hypothesis Position bias hypothesis: Examination hypothesis Examination hypothesis (WWW 07, Richardson et al.) The position bias P(e = 1 r) can be experimentally measured by presenting users with the same ad at various positions on the page, and observing the user clicks. Remark In the examination hypothesis, the position bias is modeled with the query-independent examination probability P(e r) and eliminated from the relevance estimation.

24 Depend on click pattern Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

25 Depend on click pattern Depend on click pattern: Cascade hypothesis Cascade hypothesis (WSDM 08, Carswell et al.) Cascade hypothesis assumes that an user scans each ad sequentially without any skips until she clicks on an ad and does not examine any additional ads after the click: P(e 1 = 1) = 1 P(e i = 1 e i 1 = 0) = 0 P(e i = 1 e i 1 = 1) = 1 c i 1 Novelty: The first attempt to model click pattern

26 Depend on click pattern Depend on click pattern: Cascade hypothesis Cascade hypothesis (WSDM 08, Carswell et al.) The probability of a click sequence with kth ad being clicked is: P(c = 1 r = k, q, a, u, L, S) =P(c = 1 r = k, q, a, L, S) Independence assumption k 1 =P(c = 1 r = k, q, a) P(c = 0 r = i, q, a) i=1 Cascade hypo. Remark This model is quite restrictive since it allows at most one click per query session.

27 Depend on click pattern Depend on click pattern: Multiple-click model Multiple-Click Model (WSDM 09, Guo et al.) Novelty: To enable multiple clicks in a session by incorporating a decision phase for continuing examining results. Figure: The user model of dependent click model

28 Depend on click pattern Depend on click pattern: Multiple-click model Multiple-click model (WSDM 09, Guo et al.) The probability of examination and click is given by: P(e = 1 r = 1) = 1 P(c = 1 r = i) = P(e = 1 r = i)p(c = 1 e = 1, r = i) P(e = 1 r = i + 1) = λ i P(c = 1 r = i) + P(c = 0 r = i) The probability of a click sequence with kth ad being clicked is: P(c = 1 r = k, q, a, u, L, S) =P(c = 1 r = k, q, a, L, S) Independence assumption k 1 =P(c = 1 r = k, q, a) λ i P(c = 1 r = i, q, a) + P(c = 0 r = i, q, a) i=1

29 Depend on click pattern Depend on click pattern: Dynamic Bayesian Network Dynamic Bayesian Network (WWW 09, Chapelle and Zhang) Novelty: The first attempt to model post click pattern Key idea: Model both post and perceived relevance Figure: The DBN used for clicks modeling.

30 Depend on click pattern Depend on click pattern: Dynamic Bayesian Network Dynamic Bayesian Network (WWW 09, Chapelle and Zhang) The following equations describe the model: A i = 1, E i = 1 C i = 1 P(A i = 1) = a u P(S i = 1 C i = 1) = s u C i = 0 S i = 0 S i = 1 E i+1 = 0 P(E i+1 = 1 E i = 1, S i = 0) = γ E i = 0 E i+1 = 0 where γ is the probability that an user examines the next result if she is not satisfied with the current result.

31 Depend on click pattern Depend on click pattern: Dynamic Bayesian Network Experiment: Data: 58,000,000 sessions and 682,000 unique queries from the click logs of the UK market X-axis: # of training sessions occurred at Position 1 Y-axis: MSE between the true CTRs and predicted CTRs

32 Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

33 : Temporal click model Temporal click model (SIGIR 09, Xu et al.) Key idea: (Externality) An ad may receive fewer clicks when co-displayed with high quality ads. Novelty: The first attempt to model ad externality Data study 1 Data: Ad impression sequences with exactly two ads. Two data sets are constructed by collecting one month (tens of millions) ads shown on north and south, respectively. Ground truth: Empirical CTR as the measure of ad quality. Experiment Setting: Group impressions with similar ad quality at Position 1 into one bin and plot the average CTR at Position 2, and vice versa.

34 : Temporal click model

35 : Temporal click model Data study 2 Experiment Setting: Group impressions with similar CTR at Position 1 in one bin and plot the percentage of events where the first click occurred at Position 2 and vice versa.

36 : Temporal click model Figure: The first click influenced by ad quality

37 : Temporal click model Temporal click model (SIGIR 09, Xu et al.) Positional rationality hypothesis: 1 Users examine both ads together to assess their qualities, 2 If the ad at Position 2 is much better than ad at Position 1, users would click the ad at Position 2 first

38 : Temporal click model The proposed method Input: click-through log of ad impression sequence A =< a 1, a 2 >. Output: the predicted CTR of ads. Generative process:

39 : Temporal click model Graphical model: E: examination variable, E {0, 1} R a : ad quality variable, R a [0, 1] U a : position bias variable, U a [0, 1] F: random variable for the first pick, F {a 1, a 2 } S: random variable for the re-pick, S {a 1, a 2 } C i : click random variable for ith click, C {0, 1}

40 : Temporal click model Experiments Data set: 0.3 million unique queries and 0.1 billion sessions shown at north. 1.1 million queries and 0.65 billion sessions shown at south. Evaluation: MSE between true CTRs and predicted CTRs. Baselines: 1) Naive CTR statistics (NS) estimates CTR by counting, and 2) Bayesian browsing model (BBM) (KDD 09 Liu et al.)

41 : Temporal click model

42 : Temporal click model Experiment results Both TCM and BBM are significantly better than NS for all query frequencies. TCM is noticeably better than BBM on less-frequent queries but shows similar performance on frequent ones.

43 : Relational click prediction Relational click prediction (WSDM 12, Xiong et al.) Key idea: Click events would be influenced by the similarity between co-displayed Ads. Novelty: The first attempt to model similarity influence Figure: Two ad lists for query itunes account.

44 : Relational click prediction Data study Data: 0.7 million unique queries and 0.6 million unique ads from one month click logs Experiment setting: 1 Group ads into a specific context, i.e. a triple T =< q, a, r >, where q, a and r represent query, ad and position respectively. 2 Select a triple T that appears in multiple pageviews, i.e. l =< q, ad list >. 3 Calculate similarity between ad a and other ads in each l. 4 Compute empirical CTR of T in each l, and compare them with the average CTR of T on all pageviews. Evaluation: CTR T,l = CTR T,l CTR T CTR T

45 : Relational click prediction X-axis denotes the similarity between a and other co-displayed ads. Y-axis denotes the average CTR l for different triples.

46 : Relational click prediction Data study CTR l is negatively correlated with the similarity between surrounding ads. The intuition is that: When the surrounding ads are similar to the given ad in their contents (or topics), it is likely that they will distract user s attention.

47 : Relational click prediction The proposed method Key idea: Modeling ads in an ad list together instead of treating them independently. (P(c q, a, u, L, r = 1),, P(c q, a, u, L, r = n)) T = F(X, R) where X = {x 1,, x n } includes all the feature vectors x i extracted from < q, a, u, L, r = i >, and R encodes the relation between ads.

48 : Relational click prediction Graphical model: Figure: A continuous CRF model for relational click prediction.

49 : Relational click prediction Let Y = {y 1,, y n } denotes the predicted CTRs of ads. The probability distribution of output Y conditioned on input X is defined as P(Y X) = 1 Z(X) exp h(y i, X; w) + βg(y i, y j, X) i j>i where h is the vertex feature function representing the dependence between CTR and input feature vectors, g is the edge feature function representing pairwise relationship between ads.

50 : Relational click prediction Individual modeling For simplicity, they define the vertex feature function as follows, h(y i, X; w) = (y i f (x i ; w)) 2 where f (x i ; w) is the output of any conventional click model. Relational modeling As discussed, if two ads are very similar to each other, their click probabilities will both become lower. To encode this intuition, they define the edge feature function as below. g(y i, y j, X) = s i,j (y i + y j ), where s i,j is the term similarity between ads i and j.

51 : Relational click prediction The whole model By combining all the feature functions, we obtain the overall conditional probability distribution: P(Y X) = 1 Z(X) exp (y i f (x i ; w)) 2 + βs i,j (y i + y j ) j>i i

52 : Relational click prediction Experiments Data set: 0.7 million unique queries and 0.6 million unique ads from one month click logs Extracted features: history COEC, relevance of ad to query, attractiveness of ad title and description, reputation of advertiser, etc. Baselines: Logistic Regression (LOCAL) and an variant of the proposed method with no edge features being used. Evaluation: MSE between true CTR and predicted CTR

53 : Relational click prediction The proposed method significantly outperform baselines. It performs better in lower positions than higher positions which consists with the cascade assumption. Figure: Results of NMSE.

54 : Relational click prediction They further study the performance of click prediction with respect to different levels of similarities in the ad lists. When the similarity between ads increases, the performance of CRF also increases. Figure: NMSE results at different similarity levels.

55 Depend on user intent Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

56 Depend on user intent Depend on user intent: Task-centric click model Task-centric click model (KDD 11, Zhang et al.) Novelty: The first attempt to model user behavior across multiple query sessions. Key ideas: Users tend to express their information needs incrementally, and click fresh documents that are not included before

57 Depend on user intent Depend on user intent: Task-centric click model Task-centric click model (KDD 11, Zhang et al.) Figure: The macro model of TCM.

58 Depend on user intent Depend on user intent: Task-centric click model Task-centric click model (KDD 11, Zhang et al.) Figure: The micro model of TCM.

59 Depend on user intent Depend on user intent: Task-centric click model Graphical model:

60 Depend on user intent User intent demo

61 Outline 1 An overview of web search marketing 2 An overview of click modeling 3 A survey on click models Unbiased hypothesis Position bias hypothesis Depend on click pattern Depend on user intent 4 Future work

62 Future work Click modeling for long tail query, crowdsourcing? Automatic feature construction, deep learning? Evaluation metrics Click modeling for web search in mobile device very different user browsing behavior may be totally different business model

63 Reference WWW 07, Richardson et al.: Predicting clicks: estimating the click-through rate for new ads WSDM 08, Carswell et al.: An experimental comparison of click position-bias models WSDM 09, Guo et al.: Efficient multiple-click models in web search WWW 09, Chapelle and Zhang: A dynamic bayesian network click model for web search ranking SIGIR 09, Xu et al.: Temporal click model for sponsored search WSDM 12, Xiong et al.: Relational click prediction for sponsored search KDD 11, Zhang et al.: User-click modeling for understanding and predicting search-behavior

64 Thanks

Click Chain Model in Web Search

Click Chain Model in Web Search Click Chain Model in Web Search Fan Guo 1, Chao Liu, Anitha Kannan 3, Tom Minka 4, Michael Taylor 4, Yi-Min Wang, Christos Faloutsos 1 1 Carnegie Mellon University, Pittsburgh PA 1513, USA Microsoft Research

More information

Invited Applications Paper

Invited Applications Paper Invited Applications Paper - - Thore Graepel Joaquin Quiñonero Candela Thomas Borchert Ralf Herbrich Microsoft Research Ltd., 7 J J Thomson Avenue, Cambridge CB3 0FB, UK THOREG@MICROSOFT.COM JOAQUINC@MICROSOFT.COM

More information

17.6.1 Introduction to Auction Design

17.6.1 Introduction to Auction Design CS787: Advanced Algorithms Topic: Sponsored Search Auction Design Presenter(s): Nilay, Srikrishna, Taedong 17.6.1 Introduction to Auction Design The Internet, which started of as a research project in

More information

Nominal and ordinal logistic regression

Nominal and ordinal logistic regression Nominal and ordinal logistic regression April 26 Nominal and ordinal logistic regression Our goal for today is to briefly go over ways to extend the logistic regression model to the case where the outcome

More information

Click efficiency: a unified optimal ranking for online Ads and documents

Click efficiency: a unified optimal ranking for online Ads and documents DOI 10.1007/s10844-015-0366-3 Click efficiency: a unified optimal ranking for online Ads and documents Raju Balakrishnan 1 Subbarao Kambhampati 2 Received: 21 December 2013 / Revised: 23 March 2015 / Accepted:

More information

Learning to Rank Revisited: Our Progresses in New Algorithms and Tasks

Learning to Rank Revisited: Our Progresses in New Algorithms and Tasks The 4 th China-Australia Database Workshop Melbourne, Australia Oct. 19, 2015 Learning to Rank Revisited: Our Progresses in New Algorithms and Tasks Jun Xu Institute of Computing Technology, Chinese Academy

More information

How much can Behavioral Targeting Help Online Advertising? Jun Yan 1, Ning Liu 1, Gang Wang 1, Wen Zhang 2, Yun Jiang 3, Zheng Chen 1

How much can Behavioral Targeting Help Online Advertising? Jun Yan 1, Ning Liu 1, Gang Wang 1, Wen Zhang 2, Yun Jiang 3, Zheng Chen 1 WWW 29 MADRID! How much can Behavioral Targeting Help Online Advertising? Jun Yan, Ning Liu, Gang Wang, Wen Zhang 2, Yun Jiang 3, Zheng Chen Microsoft Research Asia Beijing, 8, China 2 Department of Automation

More information

Position Auctions with Externalities

Position Auctions with Externalities Position Auctions with Externalities Patrick Hummel 1 and R. Preston McAfee 2 1 Google Inc. phummel@google.com 2 Microsoft Corp. preston@mcafee.cc Abstract. This paper presents models for predicted click-through

More information

A Logistic Regression Approach to Ad Click Prediction

A Logistic Regression Approach to Ad Click Prediction A Logistic Regression Approach to Ad Click Prediction Gouthami Kondakindi kondakin@usc.edu Satakshi Rana satakshr@usc.edu Aswin Rajkumar aswinraj@usc.edu Sai Kaushik Ponnekanti ponnekan@usc.edu Vinit Parakh

More information

Optimizing Search Engines using Clickthrough Data

Optimizing Search Engines using Clickthrough Data Optimizing Search Engines using Clickthrough Data Presented by - Kajal Miyan Seminar Series, 891 Michigan state University *Slides adopted from presentations of Thorsten Joachims (author) and Shui-Lung

More information

Revenue Optimization with Relevance Constraint in Sponsored Search

Revenue Optimization with Relevance Constraint in Sponsored Search Revenue Optimization with Relevance Constraint in Sponsored Search Yunzhang Zhu Gang Wang Junli Yang Dakan Wang Jun Yan Zheng Chen Microsoft Resarch Asia, Beijing, China Department of Fundamental Science,

More information

Basics of Statistical Machine Learning

Basics of Statistical Machine Learning CS761 Spring 2013 Advanced Machine Learning Basics of Statistical Machine Learning Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu Modern machine learning is rooted in statistics. You will find many familiar

More information

Click-Through Rate Estimation for Rare Events in Online Advertising

Click-Through Rate Estimation for Rare Events in Online Advertising Click-Through Rate Estimation for Rare Events in Online Advertising Xuerui Wang, Wei Li, Ying Cui, Ruofei (Bruce) Zhang, Jianchang Mao Yahoo! Labs, Silicon Valley United States ABSTRACT In online advertising

More information

Statistical Machine Learning

Statistical Machine Learning Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

More information

Data and Application

Data and Application Carnegie Mellon University Data and Application Tutorial of Parameter Server Mu Li! CSD@CMU & IDL@Baidu! muli@cs.cmu.edu About me Ph.D student working with Alex Smola and Dave Andersen! large scale machine

More information

A General Framework for Mining Concept-Drifting Data Streams with Skewed Distributions

A General Framework for Mining Concept-Drifting Data Streams with Skewed Distributions A General Framework for Mining Concept-Drifting Data Streams with Skewed Distributions Jing Gao Wei Fan Jiawei Han Philip S. Yu University of Illinois at Urbana-Champaign IBM T. J. Watson Research Center

More information

An Empirical Analysis of Sponsored Search Performance in Search Engine Advertising. Anindya Ghose Sha Yang

An Empirical Analysis of Sponsored Search Performance in Search Engine Advertising. Anindya Ghose Sha Yang An Empirical Analysis of Sponsored Search Performance in Search Engine Advertising Anindya Ghose Sha Yang Stern School of Business New York University Outline Background Research Question and Summary of

More information

Introduction to Hypothesis Testing. Point estimation and confidence intervals are useful statistical inference procedures.

Introduction to Hypothesis Testing. Point estimation and confidence intervals are useful statistical inference procedures. Introduction to Hypothesis Testing Point estimation and confidence intervals are useful statistical inference procedures. Another type of inference is used frequently used concerns tests of hypotheses.

More information

8. Time Series and Prediction

8. Time Series and Prediction 8. Time Series and Prediction Definition: A time series is given by a sequence of the values of a variable observed at sequential points in time. e.g. daily maximum temperature, end of day share prices,

More information

Considerations of Modeling in Keyword Bidding (Google:AdWords) Xiaoming Huo Georgia Institute of Technology August 8, 2012

Considerations of Modeling in Keyword Bidding (Google:AdWords) Xiaoming Huo Georgia Institute of Technology August 8, 2012 Considerations of Modeling in Keyword Bidding (Google:AdWords) Xiaoming Huo Georgia Institute of Technology August 8, 2012 8/8/2012 1 Outline I. Problem Description II. Game theoretical aspect of the bidding

More information

Modeling Contextual Factors of Click Rates

Modeling Contextual Factors of Click Rates Modeling Contextual Factors of Click Rates Hila Becker Columbia University 500 W. 120th Street New York, NY 10027 Christopher Meek Microsoft Research 1 Microsoft Way Redmond, WA 98052 David Maxwell Chickering

More information

Probabilistic Models for Big Data. Alex Davies and Roger Frigola University of Cambridge 13th February 2014

Probabilistic Models for Big Data. Alex Davies and Roger Frigola University of Cambridge 13th February 2014 Probabilistic Models for Big Data Alex Davies and Roger Frigola University of Cambridge 13th February 2014 The State of Big Data Why probabilistic models for Big Data? 1. If you don t have to worry about

More information

Finding Advertising Keywords on Web Pages. Contextual Ads 101

Finding Advertising Keywords on Web Pages. Contextual Ads 101 Finding Advertising Keywords on Web Pages Scott Wen-tau Yih Joshua Goodman Microsoft Research Vitor R. Carvalho Carnegie Mellon University Contextual Ads 101 Publisher s website Digital Camera Review The

More information

Lecture 2: Introduction to belief (Bayesian) networks

Lecture 2: Introduction to belief (Bayesian) networks Lecture 2: Introduction to belief (Bayesian) networks Conditional independence What is a belief network? Independence maps (I-maps) January 7, 2008 1 COMP-526 Lecture 2 Recall from last time: Conditional

More information

Purchase Conversions and Attribution Modeling in Online Advertising: An Empirical Investigation

Purchase Conversions and Attribution Modeling in Online Advertising: An Empirical Investigation Purchase Conversions and Attribution Modeling in Online Advertising: An Empirical Investigation Author: TAHIR NISAR - Email: t.m.nisar@soton.ac.uk University: SOUTHAMPTON UNIVERSITY BUSINESS SCHOOL Track:

More information

Click-through Prediction for Advertising in Twitter Timeline

Click-through Prediction for Advertising in Twitter Timeline Click-through Prediction for Advertising in Twitter Timeline Cheng Li 1, Yue Lu 2, Qiaozhu Mei 1, Dong Wang 2, Sandeep Pandey 2 1 School of Information, University of Michigan, Ann Arbor, MI, USA 2 Twitter

More information

Web and Data Analysis. Minkoo Seo June 2015

Web and Data Analysis. Minkoo Seo June 2015 Web and Data Analysis Minkoo Seo June 2015 About R user since 2011 Wrote this book Software Engineer at Google Korea http://mkseo.pe.kr/ These views are mine and mine alone and do not reflect the views

More information

Internet Advertising and the Generalized Second Price Auction:

Internet Advertising and the Generalized Second Price Auction: Internet Advertising and the Generalized Second Price Auction: Selling Billions of Dollars Worth of Keywords Ben Edelman, Harvard Michael Ostrovsky, Stanford GSB Michael Schwarz, Yahoo! Research A Few

More information

Optimizing Display Advertisements Based on Historic User Trails

Optimizing Display Advertisements Based on Historic User Trails Optimizing Display Advertisements Based on Historic User Trails Neha Gupta, Udayan Sandeep Nawathe Khurana, Tak Yeon Lee Tumri Inc. Department of Computer San Mateo, CA Science snawathe@tumri.com University

More information

Response prediction using collaborative filtering with hierarchies and side-information

Response prediction using collaborative filtering with hierarchies and side-information Response prediction using collaborative filtering with hierarchies and side-information Aditya Krishna Menon 1 Krishna-Prasad Chitrapura 2 Sachin Garg 2 Deepak Agarwal 3 Nagaraj Kota 2 1 UC San Diego 2

More information

Cell Phone based Activity Detection using Markov Logic Network

Cell Phone based Activity Detection using Markov Logic Network Cell Phone based Activity Detection using Markov Logic Network Somdeb Sarkhel sxs104721@utdallas.edu 1 Introduction Mobile devices are becoming increasingly sophisticated and the latest generation of smart

More information

Subordinating to the Majority: Factoid Question Answering over CQA Sites

Subordinating to the Majority: Factoid Question Answering over CQA Sites Journal of Computational Information Systems 9: 16 (2013) 6409 6416 Available at http://www.jofcis.com Subordinating to the Majority: Factoid Question Answering over CQA Sites Xin LIAN, Xiaojie YUAN, Haiwei

More information

REVIEW ON QUERY CLUSTERING ALGORITHMS FOR SEARCH ENGINE OPTIMIZATION

REVIEW ON QUERY CLUSTERING ALGORITHMS FOR SEARCH ENGINE OPTIMIZATION Volume 2, Issue 2, February 2012 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: A REVIEW ON QUERY CLUSTERING

More information

How to assess the risk of a large portfolio? How to estimate a large covariance matrix?

How to assess the risk of a large portfolio? How to estimate a large covariance matrix? Chapter 3 Sparse Portfolio Allocation This chapter touches some practical aspects of portfolio allocation and risk assessment from a large pool of financial assets (e.g. stocks) How to assess the risk

More information

Collaborative Filtering. Radek Pelánek

Collaborative Filtering. Radek Pelánek Collaborative Filtering Radek Pelánek 2015 Collaborative Filtering assumption: users with similar taste in past will have similar taste in future requires only matrix of ratings applicable in many domains

More information

INTRODUCTION TO CODING THEORY: BASIC CODES AND SHANNON S THEOREM

INTRODUCTION TO CODING THEORY: BASIC CODES AND SHANNON S THEOREM INTRODUCTION TO CODING THEORY: BASIC CODES AND SHANNON S THEOREM SIDDHARTHA BISWAS Abstract. Coding theory originated in the late 1940 s and took its roots in engineering. However, it has developed and

More information

Technical challenges in web advertising

Technical challenges in web advertising Technical challenges in web advertising Andrei Broder Yahoo! Research 1 Disclaimer This talk presents the opinions of the author. It does not necessarily reflect the views of Yahoo! Inc. 2 Advertising

More information

GOOGLE ADWORDS. Optimizing Online Advertising. 15.071x The Analytics Edge

GOOGLE ADWORDS. Optimizing Online Advertising. 15.071x The Analytics Edge GOOGLE ADWORDS Optimizing Online Advertising 15.071x The Analytics Edge Google Inc. Provides products and services related to the Internet Mission: to organize the world s information and make it universally

More information

Linear Threshold Units

Linear Threshold Units Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

More information

Variance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers

Variance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications

More information

Continuous Time Bayesian Networks for Inferring Users Presence and Activities with Extensions for Modeling and Evaluation

Continuous Time Bayesian Networks for Inferring Users Presence and Activities with Extensions for Modeling and Evaluation Continuous Time Bayesian Networks for Inferring Users Presence and Activities with Extensions for Modeling and Evaluation Uri Nodelman 1 Eric Horvitz Microsoft Research One Microsoft Way Redmond, WA 98052

More information

CSCI567 Machine Learning (Fall 2014)

CSCI567 Machine Learning (Fall 2014) CSCI567 Machine Learning (Fall 2014) Drs. Sha & Liu {feisha,yanliu.cs}@usc.edu September 22, 2014 Drs. Sha & Liu ({feisha,yanliu.cs}@usc.edu) CSCI567 Machine Learning (Fall 2014) September 22, 2014 1 /

More information

Internet Advertising and the Generalized Second-Price Auction: Selling Billions of Dollars Worth of Keywords

Internet Advertising and the Generalized Second-Price Auction: Selling Billions of Dollars Worth of Keywords Internet Advertising and the Generalized Second-Price Auction: Selling Billions of Dollars Worth of Keywords by Benjamin Edelman, Michael Ostrovsky, and Michael Schwarz (EOS) presented by Scott Brinker

More information

Pay Per Click Judo. Getting it Done Better and Faster by Doing What Works

Pay Per Click Judo. Getting it Done Better and Faster by Doing What Works Pay Per Click Judo Getting it Done Better and Faster by Doing What Works The Ancient Mysteries of PPC What do all those numbers in AdWords MEAN? How can I use them to create visitors of value? Leads, Sales

More information

7. Tests of association and Linear Regression

7. Tests of association and Linear Regression 7. Tests of association and Linear Regression In this chapter we consider 1. Tests of Association for 2 qualitative variables. 2. Measures of the strength of linear association between 2 quantitative variables.

More information

A Practical Application of Differential Privacy to Personalized Online Advertising

A Practical Application of Differential Privacy to Personalized Online Advertising A Practical Application of Differential Privacy to Personalized Online Advertising Yehuda Lindell Eran Omri Department of Computer Science Bar-Ilan University, Israel. lindell@cs.biu.ac.il,omrier@gmail.com

More information

Dynamical Clustering of Personalized Web Search Results

Dynamical Clustering of Personalized Web Search Results Dynamical Clustering of Personalized Web Search Results Xuehua Shen CS Dept, UIUC xshen@cs.uiuc.edu Hong Cheng CS Dept, UIUC hcheng3@uiuc.edu Abstract Most current search engines present the user a ranked

More information

Sponsored Search Ad Selection by Keyword Structure Analysis

Sponsored Search Ad Selection by Keyword Structure Analysis Sponsored Search Ad Selection by Keyword Structure Analysis Kai Hui 1, Bin Gao 2,BenHe 1,andTie-jianLuo 1 1 University of Chinese Academy of Sciences, Beijing, P.R. China huikai10@mails.ucas.ac.cn, {benhe,tjluo}@ucas.ac.cn

More information

Predictive Indexing for Fast Search

Predictive Indexing for Fast Search Predictive Indexing for Fast Search Sharad Goel Yahoo! Research New York, NY 10018 goel@yahoo-inc.com John Langford Yahoo! Research New York, NY 10018 jl@yahoo-inc.com Alex Strehl Yahoo! Research New York,

More information

Course: Model, Learning, and Inference: Lecture 5

Course: Model, Learning, and Inference: Lecture 5 Course: Model, Learning, and Inference: Lecture 5 Alan Yuille Department of Statistics, UCLA Los Angeles, CA 90095 yuille@stat.ucla.edu Abstract Probability distributions on structured representation.

More information

Predict the Popularity of YouTube Videos Using Early View Data

Predict the Popularity of YouTube Videos Using Early View Data 000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050

More information

The ABCs of AdWords. The 49 PPC Terms You Need to Know to Be Successful. A publication of WordStream & Hanapin Marketing

The ABCs of AdWords. The 49 PPC Terms You Need to Know to Be Successful. A publication of WordStream & Hanapin Marketing The ABCs of AdWords The 49 PPC Terms You Need to Know to Be Successful A publication of WordStream & Hanapin Marketing The ABCs of AdWords The 49 PPC Terms You Need to Know to Be Successful Many individuals

More information

Competition-Based Dynamic Pricing in Online Retailing

Competition-Based Dynamic Pricing in Online Retailing Competition-Based Dynamic Pricing in Online Retailing Marshall Fisher The Wharton School, University of Pennsylvania, fisher@wharton.upenn.edu Santiago Gallino Tuck School of Business, Dartmouth College,

More information

1 Maximum likelihood estimation

1 Maximum likelihood estimation COS 424: Interacting with Data Lecturer: David Blei Lecture #4 Scribes: Wei Ho, Michael Ye February 14, 2008 1 Maximum likelihood estimation 1.1 MLE of a Bernoulli random variable (coin flips) Given N

More information

Constructing Social Intentional Corpora to Predict Click-Through Rate for Search Advertising

Constructing Social Intentional Corpora to Predict Click-Through Rate for Search Advertising Constructing Social Intentional Corpora to Predict Click-Through Rate for Search Advertising Yi-Ting Chen, Hung-Yu Kao Department of Computer Science and Information Engineering National Cheng Kung University

More information

Decompose Error Rate into components, some of which can be measured on unlabeled data

Decompose Error Rate into components, some of which can be measured on unlabeled data Bias-Variance Theory Decompose Error Rate into components, some of which can be measured on unlabeled data Bias-Variance Decomposition for Regression Bias-Variance Decomposition for Classification Bias-Variance

More information

An Introduction to Information Theory

An Introduction to Information Theory An Introduction to Information Theory Carlton Downey November 12, 2013 INTRODUCTION Today s recitation will be an introduction to Information Theory Information theory studies the quantification of Information

More information

Poisson Models for Count Data

Poisson Models for Count Data Chapter 4 Poisson Models for Count Data In this chapter we study log-linear models for count data under the assumption of a Poisson error structure. These models have many applications, not only to the

More information

Seven Stages of Pay Per Click Management

Seven Stages of Pay Per Click Management Seven Stages of Pay Per Click Management Optimising the performance of your PPC Campaign PPC Management Pay per click management is increasing complex. And pay per click market categories are increasingly

More information

Interaction between quantitative predictors

Interaction between quantitative predictors Interaction between quantitative predictors In a first-order model like the ones we have discussed, the association between E(y) and a predictor x j does not depend on the value of the other predictors

More information

Question 2 Naïve Bayes (16 points)

Question 2 Naïve Bayes (16 points) Question 2 Naïve Bayes (16 points) About 2/3 of your email is spam so you downloaded an open source spam filter based on word occurrences that uses the Naive Bayes classifier. Assume you collected the

More information

Bayes and Naïve Bayes. cs534-machine Learning

Bayes and Naïve Bayes. cs534-machine Learning Bayes and aïve Bayes cs534-machine Learning Bayes Classifier Generative model learns Prediction is made by and where This is often referred to as the Bayes Classifier, because of the use of the Bayes rule

More information

The Proportional Odds Model for Assessing Rater Agreement with Multiple Modalities

The Proportional Odds Model for Assessing Rater Agreement with Multiple Modalities The Proportional Odds Model for Assessing Rater Agreement with Multiple Modalities Elizabeth Garrett-Mayer, PhD Assistant Professor Sidney Kimmel Comprehensive Cancer Center Johns Hopkins University 1

More information

Hypothesis Testing. 1 Introduction. 2 Hypotheses. 2.1 Null and Alternative Hypotheses. 2.2 Simple vs. Composite. 2.3 One-Sided and Two-Sided Tests

Hypothesis Testing. 1 Introduction. 2 Hypotheses. 2.1 Null and Alternative Hypotheses. 2.2 Simple vs. Composite. 2.3 One-Sided and Two-Sided Tests Hypothesis Testing 1 Introduction This document is a simple tutorial on hypothesis testing. It presents the basic concepts and definitions as well as some frequently asked questions associated with hypothesis

More information

Maximum Likelihood Estimation

Maximum Likelihood Estimation Math 541: Statistical Theory II Lecturer: Songfeng Zheng Maximum Likelihood Estimation 1 Maximum Likelihood Estimation Maximum likelihood is a relatively simple method of constructing an estimator for

More information

A Predictive Model for Advertiser Value-Per-Click in Sponsored Search

A Predictive Model for Advertiser Value-Per-Click in Sponsored Search A Predictive Model for Advertiser Value-Per-Click in Sponsored Search Eric Sodomka Brown University Providence, Rhode Island Sébastien Lahaie Microsoft Research New York, New York Dustin Hillard Microsoft

More information

Should Ad Networks Bother Fighting Click Fraud? (Yes, They Should.)

Should Ad Networks Bother Fighting Click Fraud? (Yes, They Should.) Should Ad Networks Bother Fighting Click Fraud? (Yes, They Should.) Bobji Mungamuru Stanford University bobji@i.stanford.edu Stephen Weis Google sweis@google.com Hector Garcia-Molina Stanford University

More information

α α λ α = = λ λ α ψ = = α α α λ λ ψ α = + β = > θ θ β > β β θ θ θ β θ β γ θ β = γ θ > β > γ θ β γ = θ β = θ β = θ β = β θ = β β θ = = = β β θ = + α α α α α = = λ λ λ λ λ λ λ = λ λ α α α α λ ψ + α =

More information

Normality Testing in Excel

Normality Testing in Excel Normality Testing in Excel By Mark Harmon Copyright 2011 Mark Harmon No part of this publication may be reproduced or distributed without the express permission of the author. mark@excelmasterseries.com

More information

Linear Classification. Volker Tresp Summer 2015

Linear Classification. Volker Tresp Summer 2015 Linear Classification Volker Tresp Summer 2015 1 Classification Classification is the central task of pattern recognition Sensors supply information about an object: to which class do the object belong

More information

Sibyl: a system for large scale machine learning

Sibyl: a system for large scale machine learning Sibyl: a system for large scale machine learning Tushar Chandra, Eugene Ie, Kenneth Goldman, Tomas Lloret Llinares, Jim McFadden, Fernando Pereira, Joshua Redstone, Tal Shaked, Yoram Singer Machine Learning

More information

17. SIMPLE LINEAR REGRESSION II

17. SIMPLE LINEAR REGRESSION II 17. SIMPLE LINEAR REGRESSION II The Model In linear regression analysis, we assume that the relationship between X and Y is linear. This does not mean, however, that Y can be perfectly predicted from X.

More information

Artificial Intelligence Mar 27, Bayesian Networks 1 P (T D)P (D) + P (T D)P ( D) =

Artificial Intelligence Mar 27, Bayesian Networks 1 P (T D)P (D) + P (T D)P ( D) = Artificial Intelligence 15-381 Mar 27, 2007 Bayesian Networks 1 Recap of last lecture Probability: precise representation of uncertainty Probability theory: optimal updating of knowledge based on new information

More information

An Introduction to Time Series Regression

An Introduction to Time Series Regression An Introduction to Time Series Regression Henry Thompson Auburn University An economic model suggests examining the effect of exogenous x t on endogenous y t with an exogenous control variable z t. In

More information

Agenda. Mathias Lanner Sas Institute. Predictive Modeling Applications. Predictive Modeling Training Data. Beslutsträd och andra prediktiva modeller

Agenda. Mathias Lanner Sas Institute. Predictive Modeling Applications. Predictive Modeling Training Data. Beslutsträd och andra prediktiva modeller Agenda Introduktion till Prediktiva modeller Beslutsträd Beslutsträd och andra prediktiva modeller Mathias Lanner Sas Institute Pruning Regressioner Neurala Nätverk Utvärdering av modeller 2 Predictive

More information

Multiple Clicks Model for Web Search of Multi-clickable Documents

Multiple Clicks Model for Web Search of Multi-clickable Documents Multiple Clicks Model for Web Search of Multi-clickable Documents Léa Laporte, Sébastien Déjean, Josiane Mothe To cite this version: Léa Laporte, Sébastien Déjean, Josiane Mothe. Multiple Clicks Model

More information

CITY UNIVERSITY OF HONG KONG. Revenue Optimization in Internet Advertising Auctions

CITY UNIVERSITY OF HONG KONG. Revenue Optimization in Internet Advertising Auctions CITY UNIVERSITY OF HONG KONG l ½ŒA Revenue Optimization in Internet Advertising Auctions p ]zwû ÂÃÙz Submitted to Department of Computer Science õò AX in Partial Fulfillment of the Requirements for the

More information

Chapter 9: Hypothesis Testing Sections

Chapter 9: Hypothesis Testing Sections Chapter 9: Hypothesis Testing Sections 9.1 Problems of Testing Hypotheses Skip: 9.2 Testing Simple Hypotheses Skip: 9.3 Uniformly Most Powerful Tests Skip: 9.4 Two-Sided Alternatives 9.5 The t Test 9.6

More information

Comparing Tag Clouds, Term Histograms, and Term Lists for Enhancing Personalized Web Search

Comparing Tag Clouds, Term Histograms, and Term Lists for Enhancing Personalized Web Search Comparing Tag Clouds, Term Histograms, and Term Lists for Enhancing Personalized Web Search Orland Hoeber and Hanze Liu Department of Computer Science, Memorial University St. John s, NL, Canada A1B 3X5

More information

Large Scale Learning to Rank

Large Scale Learning to Rank Large Scale Learning to Rank D. Sculley Google, Inc. dsculley@google.com Abstract Pairwise learning to rank methods such as RankSVM give good performance, but suffer from the computational burden of optimizing

More information

Data Mining in Web Search Engine Optimization and User Assisted Rank Results

Data Mining in Web Search Engine Optimization and User Assisted Rank Results Data Mining in Web Search Engine Optimization and User Assisted Rank Results Minky Jindal Institute of Technology and Management Gurgaon 122017, Haryana, India Nisha kharb Institute of Technology and Management

More information

The Lane s Gifts v. Google Report

The Lane s Gifts v. Google Report The Lane s Gifts v. Google Report By Alexander Tuzhilin Professor of Information Systems at the Stern School of Business at New York University, Report published July 2006 1 The Lane s Gifts case 2005

More information

Different Users and Intents: An Eye-tracking Analysis of Web Search

Different Users and Intents: An Eye-tracking Analysis of Web Search Different Users and Intents: An Eye-tracking Analysis of Web Search ABSTRACT Cristina González-Caro Pompeu Fabra University Roc Boronat 138 Barcelona, Spain cgonzalc@unab.edu.co We present an eye-tracking

More information

Tests for Two Survival Curves Using Cox s Proportional Hazards Model

Tests for Two Survival Curves Using Cox s Proportional Hazards Model Chapter 730 Tests for Two Survival Curves Using Cox s Proportional Hazards Model Introduction A clinical trial is often employed to test the equality of survival distributions of two treatment groups.

More information

Social Intelligence Report Adobe Digital Index Q2 2015

Social Intelligence Report Adobe Digital Index Q2 2015 Social Intelligence Report Adobe Digital Index Q2 2015 Key Insights Paid Social Cost per click (CPC) rates for Facebook are flat YoY while impressions fell by half and click through rates doubled 51% of

More information

Pearson's Correlation Tests

Pearson's Correlation Tests Chapter 800 Pearson's Correlation Tests Introduction The correlation coefficient, ρ (rho), is a popular statistic for describing the strength of the relationship between two variables. The correlation

More information

BALANCE LEARNING TO RANK IN BIG DATA. Guanqun Cao, Iftikhar Ahmad, Honglei Zhang, Weiyi Xie, Moncef Gabbouj. Tampere University of Technology, Finland

BALANCE LEARNING TO RANK IN BIG DATA. Guanqun Cao, Iftikhar Ahmad, Honglei Zhang, Weiyi Xie, Moncef Gabbouj. Tampere University of Technology, Finland BALANCE LEARNING TO RANK IN BIG DATA Guanqun Cao, Iftikhar Ahmad, Honglei Zhang, Weiyi Xie, Moncef Gabbouj Tampere University of Technology, Finland {name.surname}@tut.fi ABSTRACT We propose a distributed

More information

Social Media Mining. Network Measures

Social Media Mining. Network Measures Klout Measures and Metrics 22 Why Do We Need Measures? Who are the central figures (influential individuals) in the network? What interaction patterns are common in friends? Who are the like-minded users

More information

Spatial Statistics Chapter 3 Basics of areal data and areal data modeling

Spatial Statistics Chapter 3 Basics of areal data and areal data modeling Spatial Statistics Chapter 3 Basics of areal data and areal data modeling Recall areal data also known as lattice data are data Y (s), s D where D is a discrete index set. This usually corresponds to data

More information

A NURSING CARE PLAN RECOMMENDER SYSTEM USING A DATA MINING APPROACH

A NURSING CARE PLAN RECOMMENDER SYSTEM USING A DATA MINING APPROACH Proceedings of the 3 rd INFORMS Workshop on Data Mining and Health Informatics (DM-HI 8) J. Li, D. Aleman, R. Sikora, eds. A NURSING CARE PLAN RECOMMENDER SYSTEM USING A DATA MINING APPROACH Lian Duan

More information

Viewability Prediction for Online Display Ads

Viewability Prediction for Online Display Ads Viewability Prediction for Online Display Ads Chong Wang Information Systems New Jersey Institute of Technology Newark, NJ 07003, USA cw87@njit.edu Achir Kalra Forbes Media 499 Washington Blvd Jersey City,

More information

MATH4427 Notebook 2 Spring 2016. 2 MATH4427 Notebook 2 3. 2.1 Definitions and Examples... 3. 2.2 Performance Measures for Estimators...

MATH4427 Notebook 2 Spring 2016. 2 MATH4427 Notebook 2 3. 2.1 Definitions and Examples... 3. 2.2 Performance Measures for Estimators... MATH4427 Notebook 2 Spring 2016 prepared by Professor Jenny Baglivo c Copyright 2009-2016 by Jenny A. Baglivo. All Rights Reserved. Contents 2 MATH4427 Notebook 2 3 2.1 Definitions and Examples...................................

More information

Why Taking This Course? Course Introduction, Descriptive Statistics and Data Visualization. Learning Goals. GENOME 560, Spring 2012

Why Taking This Course? Course Introduction, Descriptive Statistics and Data Visualization. Learning Goals. GENOME 560, Spring 2012 Why Taking This Course? Course Introduction, Descriptive Statistics and Data Visualization GENOME 560, Spring 2012 Data are interesting because they help us understand the world Genomics: Massive Amounts

More information

College Tuition: Data mining and analysis

College Tuition: Data mining and analysis CS105 College Tuition: Data mining and analysis By Jeanette Chu & Khiem Tran 4/28/2010 Introduction College tuition issues are steadily increasing every year. According to the college pricing trends report

More information

4. Joint Distributions of Two Random Variables

4. Joint Distributions of Two Random Variables 4. Joint Distributions of Two Random Variables 4.1 Joint Distributions of Two Discrete Random Variables Suppose the discrete random variables X and Y have supports S X and S Y, respectively. The joint

More information

Estimating Sharer Reputation via Social Data Calibration

Estimating Sharer Reputation via Social Data Calibration Estimating Sharer Reputation via Social Data Calibration Jaewon Yang Stanford University jayang@stanford.edu Bee-Chung Chen LinkedIn bchen@linkedin.com Deepak Agarwal LinkedIn dagarwal@linkedin.com ABSTRACT

More information

An improved on-line algorithm for scheduling on two unrestrictive parallel batch processing machines

An improved on-line algorithm for scheduling on two unrestrictive parallel batch processing machines This is the Pre-Published Version. An improved on-line algorithm for scheduling on two unrestrictive parallel batch processing machines Q.Q. Nong, T.C.E. Cheng, C.T. Ng Department of Mathematics, Ocean

More information

Forecasting Trade Direction and Size of Future Contracts Using Deep Belief Network

Forecasting Trade Direction and Size of Future Contracts Using Deep Belief Network Forecasting Trade Direction and Size of Future Contracts Using Deep Belief Network Anthony Lai (aslai), MK Li (lilemon), Foon Wang Pong (ppong) Abstract Algorithmic trading, high frequency trading (HFT)

More information

EFFECTIVE ONLINE ADVERTISING

EFFECTIVE ONLINE ADVERTISING EFFECTIVE ONLINE ADVERTISING by Hamed Sadeghi Neshat B.Sc., Sharif University of Technology, Tehran, Iran, 2009 a Thesis submitted in partial fulfillment of the requirements for the degree of Master of

More information

Learning to Predict Subject-Line Opens for Large-Scale Email Marketing

Learning to Predict Subject-Line Opens for Large-Scale Email Marketing Learning to Predict Subject-Line Opens for Large-Scale Email Marketing Raju Balakrishnan Data Sciences, Groupon Inc. Palo Alto CA USA 94306 Email: raju@groupon.com. Rajesh Parekh Data Sciences, Groupon

More information