Scalable Hierarchical Multitask Learning Algorithms for Conversion Optimization in Display Advertising

Transcription

1 Salable Hierarhial Multitask Learning Algorithms for Conversion Optimization in Display Advertising Amr Ahmed Google Abhimanyu Das Mirosoft Researh Alexander J. Smola Carnegie Mellon University and Google ABSTRACT Many estimation tasks ome in groups and hierarhies of related problems. In this paper we propose a hierarhial model and a salable algorithm to perform inferene for multitask learning. It infers task orrelation and subtask struture in a joint sparse setting. Implementation is ahieved by a distributed subgradient orale and the suessive appliation of prox-operators pertaining to groups and subgroups of variables. We apply this algorithm to onversion optimization in display advertising. Experimental results on over 1TB data for up to 1 billion observations and 1 million attributes show that the algorithm provides signifiantly better predition auray while simultaneously being effiiently salable by distributed parameter synhronization. Categories and Subjet Desriptors G.3 [Mathematis of Computing]: Probability and Statistis; I.2.6 [Artifiial Intelligene]: Learning 1. INTRODUCTION In many ases data inferene problems do not arise in isolation. That is, we usually enounter a range of related problems and there is onsiderable benefit in solving them jointly. This insight has been exploited repeatedly and it has led to algorithms ommonly known as multitask learning tehniques [4, 22, 12, 7, 18]. Appliations, e.g. to massively multitasked spam filtering [20] show its pratial importane. The key idea is that by solving related tasks we are able to learn more about an individual task. In this paper we study the problem of onversion maximization in display advertising. That is, we fous on maximizing the ourrene of ommerially relevant ations suh as purhases, aount reation, mailing list signups, et. This involves estimating a user s propensity to perform suh ations and to identify generally suseptible populations of users. The hallenge here is that we have both a broad range work done while the authors were at Yahoo! Researh Permission to make digital or hard opies of all or part of this work for personal or lassroom use is granted without fee provided that opies are not made or distributed for profit or ommerial advantage and that opies bear this notie and the full itation on the first page. Copyrights for omponents of this work owned by others than ACM must be honored. Abstrating with redit is permitted. To opy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speifi permission and/or a fee. Request permissions from permissions@am.org. WSDM 14, February 24 28, 2014, New York, New York, USA. Copyright 2014 ACM /14/02...$ of different advertisers and also a range of subtasks (views, liks, onversions) that we wish to maximize. As is to be expeted in omputational advertising, the amount of data an be quite signifiant. Moreover, the data is not neessarily homogeneous. Tasks have wildly varying sizes, ommensurate with the finanial stake of the advertisers and the popularity of their produt. Likewise, attributes are sparse and many our only in a small number of ontexts. This requires an effetive inferene approah. Our work builds on well known multiple kernel learning [19] and ollaborative filtering tehniques namely that of effetively imposing a hyperprior on the regularization term. In doing so it is possible to ast multitask learning as a nontrivial onvex optimization problem. See e.g. [23] for details. This strategy is then ombined with a hierarhial model over task and subtask speifi parameters. Furthermore, we impose strutured sparsity along the lines of [5]. To solve the problem in pratie we rely on a distributed subgradient orale. Load-balaning is ahieved by using onsistent hashing for task distribution over proessors and distributed variable aggregation to mitigate the lateny and task restarts otherwise required in Hadoop MapRedue. That is, we use the variable distribution of [13] for storage. Subsequently we invoke a sequene of prox operators [5] to synhronize effiiently between loal and global penalties. To summarize, our ontributions are the following: We formulate the joint onversion, lik and unattributedonversion modeling problem in behavioral targeting as a large-sale hierarhial multitask learning problem and show that the onvex multitask learning approah of [23] an be adapted to this setting. We design an effiient distributed implementation of the above algorithm that sales to Terasale data. Using a real-world, web-sale display advertising targeting data set and two smaller publi datasets, we show the ability of our algorithm to signifiantly improve on the baseline modeling performane obtained by traditional single-task inferene systems. 1.1 Challenges in Conversion Maximization Reent trends in behavioral targeting and display advertising emphasize the importane of ommerially relevant ations. That is, rather than user liks, advertisers aim to maximize the number of onversions they reeive. Conversions stand for purhases, aount reation, or any other relevant ation that an advertiser sees as partiularly desirable. To obtain good performane, publishers tend to instrument

2 their websites with embedded ode whih allows third parties to apture user transations and generate user segments that are of high value for a partiular advertiser [1]. These segments primarily ontain users that are atually inlined to perform a transation as opposed to a asual visit to the web site through an aidental lik. Conversions on an advertiser s web site are either attributed to their orresponding display ads based on advertiserspeifi rules, suh as the amount of time elapsed between the onversion time and the time that the ad was last shown to the user, or they are unattributed if they annot be tied to a speifi display ad. Past work [3] has shown the superiority of targeting platforms maximizing for attributed onversions to traditional solutions maximizing for liks. For onversion-oriented behavioral targeting the traditional approah has been to only onsider attributed onversions. The orresponding inferene problem for eah advertising ampaign is then solved independently, for instane, by fitting a Support Vetor Mahine or Logisti Regression model. This generates separate models based on user data for eah ampaign. However, a typial behavioral targeting platform optimizing for attributed onversions, heneforth simply referred as onversions, faes two ore issues: There is a large volumes of user histories that need to be proessed in a periodi fashion in order to perform inferene over attributed onversions for a large number of ad ampaigns. Proessing ativities of billions of users on a daily basis imposes many hallenges suh as how to build user profiles in an effiient way, and how to optimize multiple ampaigns at the same time; When optimizing for eah ampaign separately we are likely to do poorly for infrequent ampaigns. We therefore need to design algorithms that an deal with sparseness of attributed onversions in many ampaigns. The absene of a suffiient number of labeled data for the inferene tasks reates a major bottlenek against ahieving tangible targeting performane improvement. 1.2 Multitask Learning For eah advertising ampaign, we an formulate several other related inferene tasks apart from onversion modeling: we an attempt to infer the likelihood of a lik (whih is typially a prerequisite for onversion) and to model the likelihood of unattributed onversions (the latter helps to identify similar users). These different inferene problems of eah ampaign are likely to be orrelated. Hene modeling them jointly should improve estimation performane. Additionally, it is quite likely that there is signifiant orrelations between inferene tasks aross different advertising ampaigns. For example, if there exists advertising ampaigns orresponding to different brands of ars, the onversion or lik models for all these ampaigns might be quite similar to eah other. Thus, performing a joint inferene over all these potentially-orrelated tasks together might lead to better performane than solving these inferene tasks separately. This is the basi premise of multitask learning. The key differene to onventional multitask learning is that in our ase there exists a hierarhy between tasks. That is, we expet that all tasks for a given ampaign (e.g. selling ar insurane for a partiular ompany) have related speifiity in terms of their user demographi. Hene it is only reasonable that the sets of features and preferenes are shared between them. It is to be expeted that joint feature seletion should improve the performane of eah estimator. Stritly speaking, we have two somewhat related goals one is to do well for all tasks, i.e. lik, onversion, and unattributed onversion estimation. This is a symmetri setting where the goal is to use the task orrelations to simultaneously improve the predition performane of all the tasks. The other task is to perform well for onversion predition while using the remaining data as side information. This is an asymmetri setting. While both tasks are rather related, they are subtly different in their performane riteria and in terms of the estimation problem. In this paper, we formulate two different hierarhial multitask models for these settings: a hierarhial model for the symmetri setting, and an attahment model for the asymmetri setting. In the hierarhial model, we first define an inter-ampaign orrelation matrix on a root-level set of feature weights for eah ampaign. This is then used to derive feature weights for its onversions, liks and unattributed onversions loally. In the attahment model, the interampaign orrelation matrix is applied diretly on the feature weights for the onversion model of eah ampaign. There is ample literature overing the subjet of multitask learning. However, in the ontext of our behavioral targeting problem, there are two objetives that a multitask learning algorithm should satisfy: it should be easily distributable and sale to thousands of ampaigns and millions of features; it should extend to a multi-level task hierarhy. This makes the setting rather nontrivial in terms of effiient inferene. 1.3 Approah In this paper, we use a onvex formulation approah for multitask learning. Its basi idea is desribed in several variants e.g. in the ontext of multitask learning [23], matrix inferene [16] and multiple kernel learning [14]. Essentially, one imposes a penalty over the ovariane matries governing orrelation between attribute vetors. This way we an ensure that primarily similar attribute sets and related attribute vetors are hosen. In terms of onversion maximization this means that we use an inter-ampaign ovariane matrix to model the relationships between the various ampaigns. Moreover, intraampaign ovariane matries are used to model the relationships between liks, onversions and unattributed onversions of eah ampaign. A matrix-variate normal prior is imposed on these ovariane matries. A joint optimization objetive an be formulated for all the tasks by obtaining a maximum likelihood estimate of the ovariane matries and the per-task feature weights for all the tasks. As we will show later, this objetive is onvex. It an be solved in a distributed fashion using proximal subgradient methods, suh as the Fast Iterative Shrinkage algorithm (FISTA) [6]. For both models, we use a salable alternating subspae desent method for simultaneous inferene of both the task orrelation matries and the feature weights. A key tool for ahieving salability will be to use a luster of mahines as a distributed subgradient orale [17]. Sine iterative thresholding algorithms like FISTA require a signifiant number of gradient omputations (e.g. [6] report 100 steps), our platform must preserve state and data loality between iterations. This makes it unsuitable to a naive Hadoop MapRedue implementation. Instead, we employ a onsistent hashing based synhronization algorithm. We apply our dis-

3 Ω W Y i X i i {1... m} {1... m} Ω Z W Y si X si Θ i {1... ms} s {1... m} {1... m} Figure 1: Top: Standard multitask learning using a Matrix-Variate distribution. Observations X i for ampaign reeive labels Y i. These are assigned using ampaign-speifi weight vetors W. The latter are exhangeable but not independent, hene jointly drawn from some distribution p(w Ω). Bottom: Hierarhial multitask learning. After drawing task speifi parameters Z from an exhangeable but not independent distribution we draw subtask speifi parameters W s using a joint parameter Θ and Z. The rest remains unhanged. tributed multitask learning framework to the onversion modeling problem desribed in [1] and we show how our system an improve the AUC performane signifiantly, when ompared to individual onversion modeling of eah ad-ampaign. 2. MULTITASK LEARNING 2.1 Notation We now ast the problem of ampaign-speifi estimation as a multitask learning problem. That is, we treat eah ampaign as a task. In eah suh ase we observe ovariates (patterns) x i and our goal is to infer labels y i. For simpliity we assume that x i X = R d onsists of d- dimensional vetors and moreover that y i Y are either binary Y = {±1} for lassifiation or real-valued Y = R for regression. At a later stage (in setion 3) we will assume that eah ampaign ontains a number of subtasks s. The table below gives an overview of the symbols used: X domain of observations (usually X = R d ) Y domain of labels (usually Y = {±1} or Y = R) ampaign index ( {1... m}) s sub-ampaign index (s {1... n}) j observation index (j {1... m } or j {1... m s}) x observation (x j or x sj) X set of observations for ampaign y observation (y j or y sj) Y set of labels for ampaign w parameter vetors (w or w s) W staked parameter vetors W = {... w...} z parameter vetors z for top level hierarhy Z staked parameter vetors Z = {... z...} Figure 1 aptures the formal struture of the multitask learning problem. To apture interation between ovariates x j, ampaigns and assoiated labels y j, e.g. whether a partiular user onverted on an ad in a partiular ampaign at a partiular oasion, we onsider the issue of estimating y x, for a large range of ampaigns simultaneously. We denote by m the total number of ampaigns and by m the number of observations per ampaign. Formally we onsider sets of ovariates and labels indexed by a ampaign, denoted by X = {x 1,..., x n } X and Y = {y1,..., yn } Y. Here eah pair (x j, y j) is drawn from some distribution p(x, y ) = p(y x, )p(x ) of ovariates and labels respetively. Finally, we denote by si the ombination of task, subtask, and oordinate, and by the entire vetor in the assoiated dimension. E.g. w i denotes the vetor over all subtasks s assoiated with for oordinate i. 2.2 Objetive The inferene problem is expressed either of risk minimization whenever we want to find a lassifier whih makes a small number of mistakes, or as one of maximizing the data likelihood. In the latter ase we want to find parameters W = {w 1,... w m} suh that the maximizes label likelihood: m m m p(y X, W ) = p(y X, w ) = p(y j x j, w ) (1) =1 Choies for y j x j, w are e.g. =1 j=1 p(y j x j, w ) = (2πσ 2 ) 2 1 e 1 2σ 2 (y j x j,w ) 2 (2) 1 p(y j x j, w ) = 1 + e y j x j,w (3) for regression and lassifiation respetively. A naive maximization of the onditional response likelihood p(y X, W ) leads to overfitting unless the model omplexity is overly small or unless a suitable prior is used. Multitask learning models aim to address this problem by imposing a suitable prior distribution p(w ) on W whih favors simple models and whih exploits orrelation between tasks. Consequently, instead of maximizing (1) one aims to find the Maximum-a- Posteriori (MAP) estimate of W via maximize W p(w ) m p(y X, w ) (4) =1 The hallenge is now to define models of p(w ) that are both omputationally tratable and statistially meaningful. This hierarhial modeling imperative leads to multitask learning. 2.3 Multitask Prior Our working assumption is that W is drawn from a matrixvariate distribution in suh a way as to exploit orrelations between the tasks, suh as assuming that the tasks are more onentrated in a lower dimensional subspae. This is a reasonable assumption sine there is no inherent order in whih the tasks are laid out. One option is to hoose a normal distribution as follows: W N(0, 1 d Ω) or equivalently w i N(0, Ω) (5) for all oordinates i. The likelihood of W Ω is given (up to onstants) by log p(w Ω) = tr W Ω 1 W + d log Ω + (6)

4 It is straightforward to modify this by inluding a onjugate Wishart hyperprior on Ω. Unfortunately, the outome is onave in W. An alternative is to replae the log-barrier arising from a onjugate prior on Ω by a trae onstraint and a positive semidefiniteness onstraint. That is, we replae log Ω by Ω 0 and tr Ω = 1. This is used, e.g. in [23]. Suh a modifiation leaves the eigenspae of the W -dependent part of the optimization problem unhanged. This leads to the following alternative: minimize log p(y X, w ) + λ tr W Ω 1 W (7a) W,Ω subjet to Ω 0 and tr Ω = 1 (7b) The above formulation is onvex in both W and Ω and an be solved using an effiient algorithm based on alternating subspae desent. For fixed Ω minimize (7) with respet to W. Subsequently, for fixed W, find the minimizer with respet to Ω. A simple onstrained optimization problem shows that this an be found via [ W ˆΩ W ] 2 1 = (8) tr [W W ] 1 2 This approah forms the baseline relative to whih we will ompare our proposed method. 3. HIERARCHICAL MULTITASK LEARN- ING While the flat models presented in Setion 2 an learn the orrelation struture between tasks, they are not so easily amenable for distributed optimization beause of the squared dependeny between all the tasks. Fortunately, many large sale multitask problems possess a hierarhial struture that allows us to deompose them into tasks and subtasks. For example, in display advertising eah advertiser an be regarded as a task (a ampaign) within whih we an define three subtasks as follows: Conversion predition: estimate if the user will onvert, i.e. perform a ommerially relevant ation, on the urrent display ad. Clik predition: predit if the user will lik on the urrently displayed ad. Unattributed onversion: historial data of users who onverted on previous advertisements of the advertiser. We use s to index the subtask. That is, rather than we now use the tuple s to index task and assoiated subtask, suh as (Coa Cola, liks). All remaining notation is unhanged relative to the previous setion. In a nutshell we have two options for dealing with the hierarhial struture: firstly, we estimate the joint model for all tasks, subtasks and all ampaigns. A seond strategy is to solve the model for the primary subtask of onversion estimation exlusively and to use the assoiate (seondary) subtasks only as side-information. We will refer to the former as a hierarhial model and to the latter as attahment model. The key differene is in the following assumption: Hierarhial Model: We assume that for eah task group there exists some parameter vetor z, with Z = z 1 w 11 w 12 w 13 w 11 w 12 w 13 Ω z 2 z 3 w 21 w 22 w 23 w 31 w 32 w 33 Ω w 21 w 31 w 22 w 23 w 32 w 33 z 4 w 41 w 42 w 43 w 41 w 42 w 43 Figure 2: Top: Hierarhial dependeny struture over parameter vetors for multitask learning. The intermediate parameter vetor z enapsulates ommonalities per task. Bottom: Attahment model. Here the onversion-speifi parameters are diretly oupled. For simpliity of the diagram we omitted Θ in both ases. {z 1,... z m} that speifies preferenes per task Z N(0, 1 d Ω) or equivalently z i N(0, Ω) (9) Moreover, within eah task, the distribution over subtasks is given by w i N(1 z i, Θ ). (10) This assumes that orrelations within subgroups are deoupled. Attahment Model: Denote by s = 1 the primary subtask (onversion estimation). Instead of using z as an intermediary we ouple the models diretly via w 1 and use a hierarhial model on the remaining parameters. This amounts to w 1i N(0, Ω) and w i N(1 w 1i, Θ ) for s > 1. The diagram in Figure 2 desribes the differene between both approahes for a rather simplisti struture of 4 tasks (in reality we may have millions of suh tasks). As previously disussed in Setion 2, we again resort to a reformulation that uses a trae onstraint and positive semidefiniteness rather than the log-barrier to restrit Ω and Θ. That is, instead of log p(w, Z Ω, Θ) (11) = 1 2 tr(w 1 z) (w 1 z )Θ 1 + m s log Θ tr Z Ω 1 Z + m log Ω +

5 for the hierarhial model and analogous setting for the attahment model, we use the following objetives L(W, Z, Ω, Θ): L hier (W, Z, Ω, Θ) (12) = 1 2 tr(w 1 z) (w 1 z )Θ tr Z Ω 1 Z subjet to tr Θ 0 and tr Θ = 1 and tr Ω 0 and tr Ω = 1 for the hierarhial model. Moreover, for the attahment model: L attah (W, Ω, Θ) (13) = 1 2 tr w w Θ w 1iΩ 1 w 1i 2 subjet to the same onstraints as for L hier. The only real differene is that we eliminated z and instead, we attah the model to w 1 diretly. In either ase this detahes the subtasks from the problem of joint task inferene. 3.1 Strutured Sparsity A seond aspet of multi-task learning is to use strutured sparsity [5] to selet relevant variables for an entire blok of terms jointly rather than eliminating terms for eah task individually. This is ahieved by adding a mixed norm on the parameters W and Z to the optimization problem. We need some more notation first: the p norm of a vetor x R d : x p p := i i x i p for p < and x := max x i i. Moreover, the mixed norm of a matrix X, where we apply sparsity row-wise, is defined for p, q 1 via q X p,q := X 1 p,... X d p. (14) Of partiular interest is the X 2,1 norm, whih attempts to eliminate entire rows of X at a time. Finally, we use the abbreviation X 1 := X 1,1 to denote the sum over absolute values in X. This leads to the following sparsity penalties for the hierarhial and attahment models respetively: S hier (W, Z) =λ 1 Z 1 + λ 2 Z 2,1 + (15) λ 1 W 1 + λ 2 W 2,1 S attah (W ) =λ 1 W 1 + λ 2 W 2,1 + λ 2 W 2,1 (16) The oeffiients λ 1 and λ 2 govern the trade-off between generi sparsity and group sparsity. That is, for λ 2 there will be no orrelation in sparsity patterns beyond what is obtained from data. For λ 1 = 0 we an assume that whenever any given W si 0 then also all related W s i will not vanish. 3.2 Optimization Problems We onlude this setion by stating the two optimization problems that we will solve subsequently. The key ingredients are a likelihood funtion log p(y X, W ) whih depends on the speifi problem to solve, a simplified multitask learning penalty as defined by L hier (W, Z, Ω, Θ) and L attah (W, Ω, Θ) respetively, and a sparsity penalty as in S hier (W, Z) and S attah (W ). We have the following for the hierarhial multitask model: minimize log p(y sj x sj, w s) + 1 W,Z,Ω,Θ 2 tr Z Ω 1 Z sj tr(w 1 z) (w 1 z )Θ 1 + λ 1 Z 1 + λ 2 Z 2,1 (17a) + λ 1 W 1 + λ 2 W 2,1 subjet to Ω, Θ 0 and tr Ω = tr Θ = 1. Moreover, the attahment multitask model yields: minimize log p(y sj x sj, w s) W,Ω,Θ sj tr w w Θ λ 1 W 1 + +λ 2 W 2,1 + subjet to Ω, Θ 0 and tr Ω = tr Θ = 1. i w 1iΩ 1 w 1i λ 2 W 2,1 (17b) (18a) (18b) 4. INFERENCE The optimization problems (17) and (18) are jointly onvex in (W, Z, Ω, Θ) and (W, Ω, Θ) respetively. For pratial optimization we resort to a Gauss-Southwell [15] approah of minimizing bloks of parameters at a time. In pratie this means that we alternate between minimizing with respet to W, Z and Ω, Θ. This is known to onverge to the globally optimal solution (albeit slowly on oasion). Issues of problem distribution and parallelization will be disussed in the next setion. 4.1 Covariane Updates Assume that we are given W, Z. In this ase we may find optimal values for the psd matries Ω, Θ using the derivation in (8) as follows: Ω = (tr O) 1 O (19) where O = (ZZ ) 1 2 (hierarhial) [ 1i] 12 O = w 1iw (attahment) i Likewise, for Θ we have the updates Θ = (tr T ) 1 T (20) [ ] 1 2 where T = (w i z i)(w i z i) (hierarhial) i [ 12 T = w iw i] (attahment) i This means that we an ompute T entirely with only aess to all subtask speifi parameters s for a given ampaign. Hene, as long as it is possible to have all suh data available on a single mahine, we need not ommuniate the lower level of the hierarhy outside the mahine.

6 4.2 Optimization with Sparsity Penalty Next we need to disuss update steps in terms of W and Z. Reall that we imposed a mixed norm penalty on both terms suh that we obtain group sparsity. Our strategy borrows from [5], and [10, Proposition 1]. Reall the struture of the penalties imposed by 2,1 and 1. They onstitute a hierarhy over nonzero terms in W and Z respetively the (2, 1)-norm attempts to zero out the entire set of ontributions for a given oordinate and the 1-norm ensures that even if we use an attribute, we only use it sparingly. Given Ω, Θ the remainder of the problem is a onvex unonstrained optimization problem. One of the algorithms reently proposed are of a struture resembling FISTA (Fast Iterative Suessive Thresholding) [6]. In it one interleaves a gradient desent step with regard to the onvex differentiable part of the objetive with a thresholding step with regard to the sparsity penalty. That is, for the problem: minimize f(a) + λω[a] (21) a one performs the following steps (after initializing a 0) b t+1 := a t η t af(a t) and (22) a t+1 = argmin a 1 2t t a b t λω[a] (23) Here (22) is essentially a gradient desent step in f. The step (23) is ommonly referred to as a prox-operator. The step size t i is hosen suh that t 1 i majorizes the Lipshitz onstant of af(a). We disuss omputing gradients with respet to the objetive in Setion 4.3 for the hierarhial model (the attahment model follows similarly). In this ontext we mean by F[Z, W ] either the first two lines of (17) or of (18) with Ω and Θ fixed at this point. For now note that in our ase Ω[a] deomposes into penalties applied per task. That is Ω[W, Z] = [ λ 1 z 1 + ] w s 1 + (24) s [ λ 2 z 2 + ] w s 2 s Solving (23) an be arried out for eah task and for eah z and w individually, hene it is amenable to easy distribution. Using [10, Proposition 1] one an see that performing suessive prox operations with respet to the l 1 norm and subsequently with respet to the l 2 norm lead to an exat solution of (23). For instane, for z this means that we perform the following steps z z i t i z F[Z, W ] (25) z i sgn z i max(0, z i t iλ 1) (26) and subsequently we threshold the entire vetor via z z z 2 max(0, z t iλ 2) (27) In other words, first we perform gradient desent. Then all oeffiients that are individually too small are eliminated and the remainder is shrunk. Finally, we perform shrinkage of the remainder in the diretion of their unit vetor. The objetive is either that of the hierarhial or of the attahment model. Updates with respet to W are entirely analogous and therefore omitted. tasks attributes proess for shrinkage over attributes Figure 3: The parameter matries W and Z are of size R d m. For gradient omputation we need to have aess to all parameters for a given task on a given mahine, hene the row-wise split. Subsequently, to perform shrinkage over attributes, we need all parameters pertaining to a feature (for all tasks) on a given mahine. Load balaning for both of these tasks is ahieved by onsistent hashing. 4.3 Gradients We omplete our overview of optimization by disussing the gradient in terms of W and Z. As before, we limit ourselves to a disussion of the hierarhial model of (17). Sine the reasoning required for (18) is essentially idential, we omit the details. Straightforward alulation yields zi F[Z, W ] = [ Ω 1 ] Z i (1 z i w i) (28) wsi F[Z, W ] = j + Θ 1 proess tasks independently wsi log p(y sj x sj, w s) (29) + Θ 1 (w i 1 z i) As an be seen, again all gradients deompose in terms of tasks and subtasks respetively. We will exploit this for distributed optimization. The exat form of the gradient for p(y sj x sj, w s) is straighforward to ompute for both the regression and lassifiation problem given the form of the onditional probability in (2) and (3) respetively. 5. DISTRIBUTED OPTIMIZATION We now disuss how to implement a distributed optimization algorithm effiiently on a luster of ommodity workstations. As disussed previously, invoking steps (22) and (23), and updating Ω and Θ requires the following operations: 1. Compute partial subgradients of F[Z, W ] for all ampaigns with respet to Z and W. 2. Aggregate subgradients obtained from all instanes and apply it to the model parameters. 3. Distribute oordinates (or subsets S S thereof) of the subgradients (or rather updated oordinates) to lients for appliation of the prox operator. 4. Invoke the prox operator. 5. Redistribute the results to the mahines holding the ampaign-speifi data. Sine this is an iterative proedure with two barriers per iteration (send subgradients, return values) this sounds as if it were a good fit for MapRedue. Unfortunately, this approah suffers from ineffiienies inherent in the Hadoop implementation of MapRedue: ontext in the mappers is not

7 preserved between iterations. Moreover, Hadoop ommuniates primarily via file I/O. Sine the proposed algorithm an take tens of iterations and sine we need to ommuniate parameters repeatedly, this means signifiant waste of resoures by repeatedly having to initialize the state of the mappers. Hene we resort to a method disussed in [13, 2], namely to alloate the mahines using Hadoop and then to establish an overlay ommuniation network. Algorithm 1 Distributed Optimization 1: for all i = 1 p parallel do 2: read data bloks from disk 3: for all ampaigns with m() = i do 4: ompute subgradient g 5: end for 6: write g to (key,value) store aording to m(i) 7: end for 8: reah a barrier 9: for all i = 1 p parallel do 10: read g from (key,value) store aording to m(i) 11: for all oordinates with m(j) = i do 12: solve the prox operator 13: end for 14: write to the (key,value) store aording to m() 15: Compute ontribution to suffiient statistis of Ω and write it bak to shared memory. 16: end for 17: reah a barrier 18: Read suffiient statistis of Ω and ompute new value. 5.1 Data and Task Distribution In the following we assume that we have p mahines to proess data. Reall that d denotes the number of attributes, i.e. x si R d and that m denotes the number of ampaigns. We use randomized load-balaning to determine whih mahine reeives whih portion of the data in both the databound and the parameter-bound part of the optimization proedure. This is ahieved, e.g. by onsistent hashing [11]: m() = argmin m M h(m, ) and m(i) = argmin h(m, i) (30) m M to assign mahines from a mahine pool M for ampaigns and oordinates i respetively. Finally, data exhange is arried out in the form of a distributed (key,value) store. For reasons of pratiality we used memahed as our referene implementation. This follows the design pattern of [21, 13] and it avoids file I/O for synhronization. Suh a strategy is muh more effiient than repeated invoations of Hadoop MapRedue. 5.2 Distributed Subgradient Orale By design, the subgradients of F[Z, W ] and F[W ] deompose into terms that are easily omputable in a ampaignspeifi manner (terms related to the negative log-likelihood) and terms that are easily omputable in a oordinate-speifi manner (the penalties in terms of Ω, Θ and the sparsity penalties). Furthermore, only the former requires diret aess to data, whereas the latter requires aess to a given oordinate aross all tasks. This means that we an ompute gradients in two stages: a pass over data, as performed by the workers that have the data, a reshuffle of parameters, and a finalizing pass (plus prox step) in a oordinate-speifi fashion. Likelihood gradients: Sine data is partitioned aording to tasks, subgradients with regard to w s are easily omputed via n s g s = ws log p(y sj x sj, w s). (31) j=1 s w sθ 1 w s, Next we ompute gradients with respet to 1 2 i.e. we add Θ 1 w to g s. The analogous reasoning holds for s (ws z) Θ 1 (w s z ). These gradients are then redistributed aording to Figure 3 suh that all g si for a given oordinate i (ranging over all tasks and subtasks) are available on the same mahine. Multitask gradients: At this point we an ompute oordinate speifi parts as arising from the Ω-dependent terms on a per-oordinate basis. For this purpose we only need Z i or w 1i, depending on whether we hose the hierarhial or attahment model respetively. We only need to read the weights orresponding to non-zero entries in Ω Distributed Prox Operator and Covariane Estimation The final step required is to solve the prox operator related to the 1 and 2,1 norms as these enfore sparsity. Whenever we have a fully hierarhial setting, Proposition 1 of [10] applies and we an simply perform prox steps bottom up in the proess. Whenever this assumption is not satisfied, we may still iterate the prox operator to obtain a suboptimal solution. This suffies as a desent step, sine optimization in Z and W is just a subroutine in the overall optimization senario involving Θ and Ω. Note that the prox operator an be arried out in linear time we only require omputing norms of vetors and resaling them. The data exhange is ompletely analogous to the gradient omputation, exept that we now work on attributes rather than ampaigns. After the prox operation we redistribute parameters bak into a (key,value) storage. As before, this requires a barrier to ensure that up-to-date values are available on all workers for another pass through the data. Similar to the gradient omputation phase, the read and write steps an be performed in parallel. Finally we note that estimating Θ an be done loally in eah worker however the suffiient statistis required to ompute Ω (see 19) is distributed on a per-attribute basis. Thus we overlay this step with the prox-operator step. Eah worker omputes its ontribution to the suffiient statistis using its assigned attributes. For example in the hierarhial model this redues to omputing a C C matrix i m(i) ziz i. After reahing a barrier, worker 0 then reads those partial sums and omputes the new value for Z using (8) and then writes Ω bak to a shared memory to be read by eah worker for the next iteration. Alternatively eah worker an read the suffiient statistis of Ω and ompute the new value deterministially. Moreover, instead of using (8), we ould use the graphial lasso estimation of [9] to get a sparse inverse ovariane estimation of Ω 1 from its suffiient statistis. This sparse inverse ovariane is desirable In distributed settings to minimize parameter movements when omputing the multi-task gradient as it depends on the non-zero elements of the inverse ovariane (i.e. Ω 1 ).

8 MTL Attahment MTL Hierarhial MTL Flat R2 sore Single Task AUC sore MTL Hierarhial MTL Flat Single Task Iterations Figure 4: Shool Data: R 2 Performane for various multitask learning algorithms. Note the faster onvergene and better performane of ATT-MTRL. 6. EXPERIMENTS ON PUBLIC DATA To establish the effiay of our approah we report results on both two publi datasets and one proprietary dataset. We make this hoie sine we are unable to share proprietary and finanially relevant data outside Yahoo, yet at the same time we wish to provide the reader with some means of gaining insight into the working of the proposed algorithm. In other words, we show that the proposed algorithm improves on the state of the art and simultaneously that it sales to substantial problem sizes. In terms of publi datasets we hoose two standard datasets: estimation of examination sores for students attending seondary shools in London and multi-task lassifiation of the 20-newsgroup dataset. The algorithms we ompare are the Hierarhial MTL (HIE-MTRL) and Attahment MTL (ATT-MTRL) algorithms, along with two baselines: the first one is the Single-task Learning (STL) algorithm whih does not use multitask learning, and optimizes all the tasks independently. The seond baseline is the Flat MTL (F-MTRL) algorithm of [23] that uses a matrix-variate normal prior on the task orrelation matrix. This algorithm performs multitask learning, however it does not aount for the hierarhial task and subtask struture. It flattens the task hierarhy and treats (task, subtask) as individual tasks and learns a joint ovariane struture. 6.1 Student sore estimation This dataset has been used widely for studying multitask regression 1. It onsists of the exam sores of 15, 362 students from 139 seondary shools in London during 1985, 1986 and Originally, the input onsists of the year of the exam, four shool-speifi and three student-speifi attributes. The goal is to estimate the exam sores for the students. Several papers [23, 4] evaluate multitask learning by treating the shool-ids as attributes, but one ould arguably treat this dataset as speifying a hierarhial task/subtask struture, where the shool ID refer to tasks and the exam years for eah shool orrespond to the subtasks of the shool ID. Thus, there are a total of 139 tasks, with up to 3 sub Iterations Figure 5: 20-newsgroup MTL: AUC Performane for various multitask learning algorithms. The hierarhial MTL algorithm starts off better than the flat algorithm and it onsistently outperforms both single task and flat multitask learning. tasks for eah task. We note that several tasks have only 2 tasks in the dataset. We replae eah ategorial attribute with one binary variable for eah possible attribute value as in [4] but remove the attributes orresponding to the exam years. As a result of this preproessing, we have a total of 24 input attributes. We use a 66/34 split of the dataset to use as training instanes and test instanes, and report the average (over all years and shools) performane of our algorithms, on the test set. For our performane measure, we use the normalized inner produt between the input sore vetor and the predited sore vetor. This measure is proportional to the squared multiple orrelation oeffiient R 2, a normalized version of the regression error, defined as R 2 := 1 i (yi fi)2. (32) Var[y] That is, it is the ratio between explained variane and total variane. See e.g. [8] for further details. For F-MTRL, HIE-MTRL and ATT-MTRL we use 5-fold ross validation to determine the optimal value of the appropriate regularization onstants and learning rate. Figure 6.1 plots the R 2 performane of the algorithms as optimization progresses. As an be seen in the graph, and as also reported by [23, 4], multitask learning provides a signifiant performane improvement. F-MTRL improves over the baseline by around 5%. However, by using the task-subtask hierarhy, our HIE-MTRL and ATT-MTRL obtain a further improvement from a sore of around 0.71 (for F-MTRL) to almost The performane for ATT-MTRL was slightly better than HIE-MTRL in this dataset. 6.2 Multi-Task Classifiation The task at hand is multi-task lassifiation of the 20- newsgroup dataset 2. The goal here is to predit the newsgroup of a given post. The 20 news groups are arranged into a two-level hierarhy. The first level omprises 5 ategories: 2 jason/20newsgroups/ 25

9 Table 1: 20-newsgroup analysis: AUC performane for single task and multitask algorithms on varying perentages of the data. Fration of data STL F-MTRL HIE-MTRL 20% % % % % politis, religion, rereational, siene, and omputers. Eah ategory has 2-5 subategories in the seond level. We map ategories to tasks and sub-ategories to sub-tasks. The dataset omprises 18k douments and we followed the standard test/train split. We removed stop words and words appearing less than 10 times. Five-fold ross validation is used to determine the values of the regularization parameters for all models. We measure lassifiation auray using the AUC measure. Sine in this dataset we do not have the notion of an anhor task, we only use the symmetri HIE-MTRL formulation. In Figure 5 we ompare the performane of HIE- MTRL against the state of the art algorithm in [23] F-MTRL and against single-task baseline STL. As evident from figure HIE-MTRL improves over the F-MTRL baseline by around 2% points and the improvement was statistially signifiant. Moreover HIE-MTRL outperforms all other ompeting algorithms in terms of speed of onvergene. To see the effet of varying the training set size, we selet different frations of the data for eah task to form the training set while keeping the test set fixed. As an be seen in Table 1 HIE-MTRL outperforms all other ompeting algorithms for a range of different sample sizes. Moreover, the improvement of HIE-MTRL over ompeting algorithms is more apparent when the training data size is small (a 5% improvement over the F-MTRL baseline using a 20% fration of the training data). 7. OPTIMIZING DISPLAY ADVERTISING 7.1 Data We olleted 4 weeks of advertising data, i.e. impressions, liks, and onversions, for a total of 1, 468 advertising ampaigns. 3 Eah ampaign is treated as a separate targeting task. 66% of the data is used for training, while the remaining 34% is used for soring. The train/test split is performed using a referene time stamp (impressions before that time stamp used for training and afterwards impression for testing). Sine the user profiles span 56 days of user history, eah training/soring example is preeded by at least 4 weeks of user events. This benhmark data set enables us to perform rigorous offline experiments. We ount users based on the unique number of browser ookies (see table below). days users features ampaigns dataset size , TB We study the performane of our tehniques ompared to the baseline system developed in [3]. We mainly ompare modeling performane in terms of the area under the ROC urve (AUC). Unless otherwise speified, all metris are measured as onversion-weighted average of AUC aross all ampaigns 3 We note here that data from users that opted out of behavioral targeting were not olleted. in the benhmark set. We denote the onversion-weighted average of AUC as Weighted AUC. We represent eah user using features from both ative and passive observations. Passive observations inlude viewing ads and visiting pages in whih an ation is not speifially required upon seeing the page. Ative observations inlude issuing searh queries and liking ads in whih users atually perform an ation on the page. Eah advertising ampaign has three subtasks: Prediting onversions: This sub task ontains data that shows whether users onverted on a given ampaign. That is, it ontains information whether they performed an advertiser-speified ation suh as purhasing a produt or filling a form. Prediting liks: This subtask ontains data that shows whether users liked on the ad of this ampaign or not. Predition on auxiliary (unattributed onversion) data: This subtask ontains data that shows whether users onverted on histori data on related ampaigns of the same advertiser. This data is supplied by the advertiser. We define the feature weight for a given user-(sub)ampaign example to be the number of days (before showing the user the ampaign ad) in whih the feature appears. Our platform experienes a large variane of feature weights aross our feature types thus making it hard to set a single ountthreshold below whih we onsider the feature to be irrelevant. We thus rely on the learning algorithm to perform joint onversion optimization and feature seletion. 7.2 Results All experiments reported in this setion were performed using 300 mahines. We assess the performane both in terms of AUC auray and salability of the algorithms. The attahment multitask learning algorithm (ATT-MTRL) signifiantly outperforms flat multitask and single task learning. Moreover, the results for hierarhial multitask learning (HIE-MTRL) were only slightly inferior to ATT-MTRL (thus we omit them for spae limitations). This finding is onsistent with our findings of Setion 6.1. Note that ATT- MTRL also performs multitask feature seletion, whih is essential here due to the large feature spae. We ompare our performane with the baseline Single-task Learning, whih optimizes for all the tasks and subtasks separately. We omit omparing with the F-MTRL for this task sine the flat MTL requires flattening the task-subtask struture ( 2k tasks) whih results in massive weight vetor movements aross mahines and as suh does not sale to this dataset (though in Table 4 we show the effet of introduing task and subtask ovariane on the overall performane). The parameters for all models were tuned on a validation set. In Table 2 we report the overall performane of the model against the baseline. As we an see, our model learly outperforms the baseline. All improvements of our models over the baselines are statistially signifiant. Note that the task of onversion predition is very diffiult sine positive examples are very rare. Seondly, we quantify the effet of feature seletion. For this purpose we selet the top 10k, 30k and 50k features (using mutual-information measure) and use them in the STL. For omparison we run ATT-MTRL using onservative

10 Table 2: Attahment multitask performane. AUC STL ATT-MTRL all subtasks onversions auxiliary (unattributed) liks Table 3: Feature seletion effetiveness: Conversion AUC features STL + l 2 + top features ,000 STL + l 2 + top features ,000 STL + l 2 + top features ,000 ATT-MTRL (aggressive) ,992 ATT-MTRL (onservative) ,789 Table 4: Ablation study for ATT-MTRL. AUC onversions all sub-tasks L L1+L L1+L12+Θ L1+L12+Θ+Ω (λ 1 = 0.4, λ 2 = 10) and aggressive (λ 1 = 0.4, λ 2 = 25 ) feature seletion parameters. The results in Table 3 are reported in terms of the weighted average AUC measure. Finally, in table 4 shows the ontributions of the various omponents of the ATT-MTRL algorithm towards the learning performane. Ω refers to the task-orrelation regularization, Θ refers to the sub-task orrelation regularization, and L1 and L12 refer to per-ampaign and multitask feature seletion respetively. As seen from the figure, using multitask feature seletion(l12) leads to only a marginal improvement over single task feature seletion(l1). However, adding the multitask learning omponents lead to a signifiant improvement over using just L1 and L12 regularization. These results learly show the importane of leveraging ross-ampaign and ross-ampaign-subtask information to improve the performane of ampaigns with very few onversions, as opposed to the baseline tehniques. 8. CONCLUSION In this paper we addressed the problem of hierarhial multitask learning when tasks are organized in a hierarhy. We presented two onvex formulations to this problem and showed that models that exploit the hierarhial struture outperformed flat models. Furthermore, we showed how to sale our models to a tera-sale advertising task. An advantage of our hierarhial formulation is the utilization of the task substruture for effiient parameter distribution that redues parameter movements aross mahines. We validated our models on both publi and private datasets with favorable performane. 9. REFERENCES [1] A. Ahmed, M. Aly, A. Das, A. Smola, and T. Anastasakos. Web-sale multi-task feature seletion for behavioral targeting. In CIKM, [2] A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. Smola. Salable inferene in latent variable models. In Web Siene and Data Mining (WSDM), [3] M. Aly, A. Hath, V. Josifovski, and V. K. Narayanan. Web-sale user modeling for targeting. In WWW, [4] A. Argyriou, T. Evgeniou, and M. Pontil. Convex multi-task feature learning. Mahine Learning, 73(3): , [5] F. Bah, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-induing penalties. Foundations and Trends in Mahine Learning, 4(1):1 106, [6] A. Bek and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sienes, 2(1): , [7] R. Caruana. Multitask learning. Mahine Learning, 28:41 75, [8] N. R. Draper and H. Smith. Applied Regression Analysis. John Wiley and Sons, New York, NY, [9] J. Friedman, T. Hastie, and R. Tibshirani. Sparse inverse ovariane estimation with the graphial lasso. Biostatistis, 9(3): , [10] R. Jenatton, J. Mairal, G. Obozinski, and F. Bah. Proximal methods for hierarhial sparse oding. Journal of Mahine Learning Researh, 12: , [11] D. Karger, E. Lehman, T. Leighton, M. Levine, D. Lewin, and R. Panigrahy. Consistent hashing and random trees: Distributed ahing protools for relieving hot spots on the world wide web. In Symposium on the Theory of Computing STOC, pages , New York, May Assoiation for Computing Mahinery. [12] T. R. Shultz and F. Rivest. Using knowledge to speed learning: A omparison knowledge-based asade-orrelation and multi-task learning. In Pro. Intl. Conf. Mahine Learning, pages Morgan Kaufmann, San Franiso, CA, [13] A. J. Smola and S. Narayanamurthy. An arhiteture for parallel topi models. In Very Large Databases (VLDB), [14] S. Sonnenburg, G. Rätsh, C. Shäfer, and B. Shölkopf. Large sale multiple kernel learning. Journal of Mahine Learning Researh, 7: , [15] W. H. Southwell. Fitting data to nonlinear funtions with unertainties in all measurement variables. Comput. J., 19(1):69 73, [16] N. Srebro and A. Shraibman. Rank, trae-norm and max-norm. In P. Auer and R. Meir, editors, Pro. Annual Conf. Computational Learning Theory, number 3559 in Leture Notes in Artifiial Intelligene, pages Springer-Verlag, June [17] C. Teo, Q. Le, A. J. Smola, and S. V. N. Vishwanathan. A salable modular onvex solver for regularized risk minimization. In Pro. ACM Conf. Knowledge Disovery and Data Mining (KDD). ACM, [18] S. Thrun and J. O Sullivan. Disovering struture in multiple learning tasks: the TC algorithm. In Pro. Intl. Conf. Mahine Learning, pages Morgan Kaufmann, [19] M. Varma and B. R. Babu. More generality in effiient multiple kernel learning. In A. P. Danyluk, L. Bottou, and M. L. Littman, editors, ICML, volume 382 of ACM International Conferene Proeeding Series, page 134. ACM, [20] K. Weinberger, A. Dasgupta, J. Attenberg, J. Langford, and A. J. Smola. Feature hashing for large sale multitask learning. In L. Bottou and M. Littman, editors, International Conferene on Mahine Learning, [21] J. Ye, J. Chow, J. Chen, and Z. Zheng. Stohasti gradient boosted distributed deision trees. In CIKM. ACM, [22] K. Yu, V. Tresp, and A. Shwaighofer. Learning gaussian proesses from multiple tasks. In Proeedings of the 22nd International Conferene on Mahine Learning, volume 119, pages ACM, [23] Y. Zhang and D.-Y. Yeung. A onvex formulation for learning task relationships in multi-task learning. In Unertainty in Artifiial Intelligene, 2010.