Recommending Questions Using the MDL-based Tree Cut Model

Transcription

1 WWW 2008 / Refereed Trak: Data Mining - Learning April 2-25, 2008 Beijing, China Reommending Questions Using the MDL-based Tree Cut Model Yunbo Cao,2, Huizhong Duan, Chin-Yew Lin 2, Yong Yu, and Hsiao-Wuen Hon 2 Shanghai Jiao Tong University, Shanghai, China, {summer, yyu}@apex.sjtu.edu.n 2 Mirosoft Researh Asia, Beijing, China, {yunbo.ao, yl, hon}@mirosoft.om ABSTRACT The paper is onerned with the problem of question reommendation. Speifially, given a question as query, we are to retrieve and rank other questions aording to their likelihood of being good reommendations of the queried question. A good reommendation provides alternative aspets around users interest. We takle the problem of question reommendation in two steps: first represent questions as graphs of topi terms, and then rank reommendations on the basis of the graphs. We formalize both steps as the tree-utting problems and then employ the MDL (Minimum Desription Length) for seleting the best uts. Experiments have been onduted with the real questions posted at Yahoo! Answers. The questions are about two domains, travel and omputers & internet. Experimental results indiate that the use of the MDL-based tree ut model an signifiantly outperform the baseline methods of word-based VSM or phrasebased VSM. The results also show that the use of the MDL-based tree ut model is essential to our approah. Categories and Subjet Desriptors H.3.3 [Information Storage and Retrieval]: Information Searh and Retrieval searh proess; H.4.m [Information Systems and Appliations]: Misellaneous - BSP; I.7.m [Doument and Text Proessing]: Misellaneous General Terms Algorithms, Experimentation, Human Fators Keywords Question Reommendation, Query Suggestion, Tree Cut Model, Minimum Desription Length. INTRODUCTION Community-based Q&A servie (referred to as QA) is a kind of web servie where people an post questions and answer other people s questions. The growth in QA has been one of the emerging trends at the age of Web 2.0. The typial examples provided by the ommerial searh engines inlude Yahoo! Answers, Live QnA 2, and Baidu Zhidao 3. The huge number of retail and business sites that provide FAQ servies an be viewed as the same type of system. Over time, QA servies build up very large arhives of previous questions and their answers. In order to avoid the lag time involved with waiting for a personal response, a QA servie will typially automatially searh this arhive to see if the same question has previously been asked. If the question is found, then a previous answer an be provided with very little delay. That is what we alled question searh. For example, given the question in the first row of Table, question searh is to return the question in the seond row, whih are semantially equivalent to the input question and expeted to have the same answer. Many methods have been proposed for takling the problem [3-5]. Table : Question searh vs. question reommendation Queried question: Any ool lubs in Berlin or Hamburg? Question searh: What are the best/most fun lubs in Berlin? Question reommendation How far is it from Berlin to Hamburg? Where to see between Hamburg and Berlin? Hong long does it take to get to Hamburg from Berlin on the train? Cheap hotel in Hamburg? In this paper, to omplement question searh, we are to explore a novel appliation whih we all question reommendation. We onsider a question as a ombination of question topi and question fous. Question topi usually presents the major ontext/onstraint of a question (e.g., Berlin, Hamburg) whih haraterizes users interest. Question fous (e.g., ool lub) presents ertain aspet (or desriptive features) of the question topi. When users ask questions, they usually are pretty lear about their question topis. However, they might not be aware that there exist several aspets around the question topis that are also worth exploring. Thus, it is desirable that an automati system an suggest alternative aspets of the queried question topi and reommend the related questions Copyright is held by the International World Wide Web Conferene Committee (IW3C2). Distribution of these papers is limited to lassroom use, and personal use by others. WWW 2008, April 2 25, 2008, Beijing, China. ACM /08/

2 WWW 2008 / Refereed Trak: Data Mining - Learning April 2-25, 2008 Beijing, China aordingly. For example, if a user wants to leave for Berlin and Hamburg, it will be very useful that the system an suggest the question where to see between Hamburg and Berlin? (or others in the third row of Table ) although her or his question is any ool lubs in Berlin or Hamburg? To the best of our knowledge, no existing study has been presented with regard to question reommendation. Query suggestion an be onsidered as an analog of question reommendation in the setting of web searh. However, it is not designed for long queries like questions. We takle the problem of question reommendation in two steps: first represent questions as graphs of topi terms, and then rank reommendation andidates on the basis of the graphs. To represent question as graphs of topi terms, we need have a method for the extration of topi terms. We onsider as topi terms all the terms that an be used to represent the question topi or the question fous. For example, Hamburg, Berlin, and ool lub an be extrated as topi terms from the query in Table. We an make use of all the noun phrases and ngrams as the andidate topi terms. However, the number of noun phrases and ngrams is usually too large to manage and some of them are too sparse to be aurately modeled. Therefore, we propose to redue some topi terms to their prefixes or suffixes. For example, given two topi terms embassy suite hotel and suite hotel, we may remove embassy suite hotel from the set of topi terms by onsidering it same as suite hotel. We an all this redution of topi terms. The redution an help redue the size of the voabulary of topi terms and moderate the sparseness problem. A good reommendation provides alternative aspets (question fous) around users interest (question topi). That is to say, a good reommendation should differ from the queried question in question fous and stik to the queried question in question topi. Thus, provided that questions are represented by topi terms, the step of ranking reommendations should be able to disriminate question topi from question fous. The disrimination problem an be omplex beause it should be based on all the related questions as well as the queried question. For the question in Table, for example, the topi terms Hamburg and Berlin are onsidered more likely to be question topis than the topi term ool lub, onsidering that, in the six questions in Table, the former two topi terms ours muh more than the latter one. In this paper, we propose using the MDL-based (Minimum Desription Length) tree ut model for handling the two issues raised above, redution of topi terms and disrimination between question topi and question fous. MDL is a priniple of data ompression and statistial estimation from information theory, whih enables the proposed approah to optimize balane between speifiity and generality. We empirially ondut the question reommendations with the questions about travel and the questions about omputers & internet. Both two kinds of questions are from Yahoo! Answers. Experimental shows that our proposed approah using the MDLbased tree-ut model an signifiantly outperform the baseline methods of the word-based VSM (Vetor Spae Model) and the phrase-based VSM. The word-based VSM is just the onventional VSM [22] for information retrieval in whih words are used to represent queries and douments (in our ase, they are questions). The phrase-based VSM differs from the word-based VSM in that it uses the extrated topi terms to represent questions. The ontribution of this paper an be summarized as, To the best of our knowledge, this paper is the first effort of question reommendation. As will be illustrated in Setion 2, the problem of question reommendation is different from query suggestion/substitution of the web searh. The MDL-based tree ut model is proposed to takle two important issues regarding question reommendations. The two issues are redution of topi terms and disrimination between question topi and question fous. The MDL-based approah enables the question reommendation to optimize balane between speifiity and generality. Extensive experiments have been onduted to evaluate the proposed approah using a large olletion of real questions provided at Yahoo! Answers. The rest of the paper is organized as follows: Setion 2 introdues the related work. In Setion 3, we formalize the problem of question reommendation. In Setion 4, we present our MDLbased approah to question reommendation. In Setion 5, we empirially verify the effetiveness of the proposed approah. Setion 6 summarizes our work and disusses the future work. 2. RELATED WORK 2. Question Searh As shown in Table, question searh is the researh area most related to question reommendation sine it also deals with questions as targets diretly. Given a queried question, question searh is to find the questions that are semantially similar to the queried question. The major fous of the researh is to takle the lexial hasm problem between questions. The researh of question searh is first onduted using FAQ data [3, 8, 23]. FAQ Finder [3] heuristially ombines statistial similarities and semanti similarities between questions to rank FAQs. Conventional vetor spae models are used to alulate the statistial similarity and WordNet [8] is used to estimate the semanti similarity. Sneiders [23] proposed template based FAQ retrieval systems. Lai et al. [8] proposed an approah to automatially mine FAQs from the Web. However, they did not study the use of these FAQs after they were olleted. Reently, the researh of question searh has been further extended to the QA. For example, Jeon et al. [3-5] ompared four different retrieval methods, i.e. osine similarity, Okapi, language model (LM), and statistial mahine translation model (SMT), for automatially fixing the lexial hasm between questions of question searh. They found that the SMT-based method performed the best. Atually, there is a bar between searh and reommendation. If we want to have lose question fous as well as question topi, it is more suitable to ahieve that with question searh; if we want to have further question fous around ertain question topi, it is better to leverage question reommendation. 2.2 Query Suggestion / Substitution In the setting of web searh, there are two researh areas related to the problem of question reommendation, namely query suggestion and query substitution. Query suggestion is a funtionality to help users of a searh engine to better speify their information need by suggesting related queries that has been frequently used by other users. The suggestions usually onsist of synonymous queries and relevant 82

3 WWW 2008 / Refereed Trak: Data Mining - Learning April 2-25, 2008 Beijing, China queries. Thus, query suggestion is atually an analog of question searh as shown in Table in the setting of web searh. In ontrast to that, the suggestions provided by question reommendation may not be synonymous to the queried question. Typial methods for query suggestion exploit query logs [7, 9, 2, 24] and doument olletions [, 7], by assuming that in the same period of time, many users share the same or similar interests, whih an be expressed in different manners. However, none of the methods is aimed at handling the sentene-level reommendation as question reommendation does. Query substitution [6] replaes a user's original searh query by generating a new query ontaining terms losely related to the original query. To ompare that, as will be elaborated later, question reommendation is onduted to explore aspets whih diverge from the original (question) query. 3. PROBLEM STATEMENT Given an initial searh query of a question q, we wish to reommend a question q suh that q and q reflet different aspets of users interests. We do this by substituting appropriate phrases as shown shematially in Figure. Any ool lubs in Berlin or Hamburg? Hamburg Berlin heap hotel ool lub where to see how far how long does it take Where to see between Hamburg and Berlin? How far is it from Berlin to Hamburg? How long does it take to Hamburg from Berlin on the train? Cheap hotel in Hamburg? Figure. An example on question reommendation We onsider a question as a ombination of question topi and question fous. Question topi usually presents the major ontext/onstraint of a question (e.g., Berlin, Hamburg) whih haraterizes users interest. Question fous (e.g., ool lub) presents ertain aspet (or desriptive features) of the question topi. As for question reommendation, we are to substitute the question fous so that users an explore different aspets of their interests by reading reommendations. In Figure, we assume that there exists a tree (graph) of topi terms representing the queried question and targeted questions. In the tree, the nodes representing question topi are expeted to be loser to the root node than the nodes representing question fous. That is to say, the nodes representing question fous tend to appear as leaf nodes. Thus, as for question reommendation, we an then substitute the topi terms by beginning at the leaf nodes and stopping at ertain level of the tree. We all the tree representing questions as question tree. In order to ahieve the substitution-based reommendation as Figure exemplifies, we need to solve the following two sub-problems: Representing questions as trees (graphs) of topi terms: Regarding this, we have two questions to answer: ) how do we build the voabulary of topi terms suh that the voabulary well models both the given questions and the new queried question? 2) Given the voabulary of topi terms, how do we onstrut a question tree as that in Figure? Ranking of reommendation andidates Regarding this, we have also two questions to answer: ) how do we disriminate the question fous from the question topi so that we an substitute the question fous for exploring different aspets of users interest? Given a tree as presented in Figure, this question is equivalent to that of hoosing a ut indiated by the dash line. 2) Based on a ut of question tree, how do we rank various hoies of substitution? For example, in Figure, the possible substitutions inlude where to see, how far, how long does it take, and heap hotel. In the next setion, we are to explain our MDL-based approah to handling the issues raised by the questions above. 4. QUESTION RECOMMENDATION Our approah to question reommendation onsists of two steps: represent questions as trees (graphs) of topi terms and then rank reommendation andidates. In this setion, we are to explain the two steps in details. As our approah involves muh use of a MDL-based tree ut model, we will first introdue what a MDL-based tree ut model is before the detailed explanation of the two steps of our approah. 4. Tree Cut Model and MDL Formally, a tree ut model M [9] an be represented by a pair onsisting of a tree ut Γ, and a probability parameter vetor θ of the same length, that is, where Γ and θ are M = ( Γ, θ ) () Γ = [ C, C2,... Ck ], θ = [ p( C), p( C2 ),... p( Ck )] (2) where C, C2,... C k are lasses determined by a ut in the tree and k = p( C = i i ). A ut in a tree is any set of nodes in the tree that defines a partition of all the nodes, viewing eah node as representing the set of hild nodes as well as itself. For example, the ut indiated by the dash line in Figure 2 orresponds to three lasses: [ n, n 0 ], [ n 2, n2, n22, n23], and [ n, n 3 24 ]. In the next two sub-setions, eah node represents a topi term. n 0 n n n 2 3 n2 n 22 n n Figure 2. An example on the tree ut model A straightforward way for determining a ut of a tree is to ollapse the nodes of less frequeny into its parent node. However, the method is too heuristi for it relies muh on manually tuned frequeny threshold. In our pratie, we turn to use a theoretially 83

4 WWW 2008 / Refereed Trak: Data Mining - Learning April 2-25, 2008 Beijing, China well-motivated method based on the MDL (Minimum Desription Length) priniple. MDL is a priniple of data ompression and statistial estimation from information theory [2, 20, 2]. Given a sample S and a tree ut Γ, we employ MLE to estimate the parameters of the orresponding tree ut model M ˆ = ( Γ, ˆ θ), where θˆ denotes the estimated parameters. Aording to the MDL priniple, the desription length [9] L ( Mˆ, S) of the tree ut model Mˆ and the sample S is the sum of the model desription length L (Γ), the parameter desription length L ( ˆ θ Γ), and the data desription length L ( S Γ, ˆ θ ), i.e. L ( Mˆ, S) = L( Γ) + L( ˆ θ Γ) + L( S Γ, ˆ) θ (3) The model desription length L (Γ) is a subjetive quantity whih depends on the oding sheme employed. Here, we simply assume that eah tree ut model is equally likely a priori. The parameter desription length L ( ˆ θ Γ) is alulated as k L( ˆ θ Γ) = log S (4) 2 where S denotes the sample size and k denotes the number of free parameters in the tree ut model, i.e. k equals the number of nodes in Γ minus one. The data desription length L ( S Γ, ˆ θ ) is alulated as L ( S Γ, ˆ) θ = log pˆ( n) (5) n S where f ( C) (6) pˆ ( n) = C S where C denotes the lass of node n; f (C) denotes the total frequeny of nodes in lass C in the sample S. With the desription length defined as (3), we wish to selet a tree ut model with the minimum desription length and output it as the result of redution. Note that the model desription length an be ignored beause it is same for the tree ut models. The MDL-based tree ut model was originally introdued for handling the problem of generalizing ase frames using a thesaurus [9]. To the best of our knowledge, no existing work utilized it for question reommendation, or even query suggestion/substitution. This may be partially beause of the unavailability of the resoures (e.g., thesaurus) whih an be used for embodying the queries/questions in a tree struture. In the next two sub-setions, we will explain our methods for onstruting the tree struture of topi terms and questions. 4.2 Representing Questions as Trees (Graphs) of Topi Terms In our view, the problem of representing questions as trees of topi terms involves two issues: (a) aquiring topi terms; and (b) linking topi terms. We will elaborate our methods used to takle the two issues in the following subsetions Aquiring Topi Terms The topi term aquisition proess onsists of two phases: extration of topi terms and redution of topi terms. Extration of Topi Terms In general, there are many possible hoies of linguisti units suh as words, noun phrases, and n-grams, whih an be used to representing topis. A good topi term should, together with other topi terms, apture the major meaning of a sentene and distinguish one topi from others. Words are usually too speifi to outline the major meaning of sentenes. Therefore, in our pratie, the extration of topi terms only onsiders noun phrases and n-grams as andidates of topi terms. BaseNP: A base noun phrase (BaseNP) is defined as a simple and nonreursive noun phrase [4]. In many ases Base NPs represent holisti and no-divisible onepts, and thus we extrat BaseNPs (instead of noun phrases) as topi term andidates. The BaseNPs inlude both the multi-word terms (e.g., budget hotel, nie shopping mall) and the named entities (e.g., Berlin, Hamburg, Forbidden City). We make use of the tool introdued in [25] to extrat the BaseNPs. WH-ngram: Ngram [5] of words an also be used as the topi term andidates. However, most meaningful n-grams are already overed by the BaseNPs. Thus, to omplement the BaseNPs, we onsider only using the WH-ngrams, whih are the ngrams beginning with the WH-words. The WH-words inlude when, what, where, whih, and how. Many ngrams (e.g., where to ) are noisy terms. There are many methods [, 6] for testing whether an ngram is a meaningful olloation/phrase. In our pratie, we make use of the MDLbased tree ut model to eliminate the noisy WH-ngrams. Table 2. The examples on topi terms Type Topi Term Frequeny hotel 3983 suite hotel 3 embassy suite hotel nie suite hotel 2 western hotel 40 BaseNP good western hotel 4 inexpensive western hotel 2 beahfront hotel 5 good beahfront hotel 3 great beahfront hotel 3 nie hotel 224 affordable hotel 48 where 365 where to learn 6 where to learn omputer WH-ngram where to learn Japanese where to buy 5 where to buy ginseng where to buy insurane 23 where to buy tea 2 Table 2 provide the example BaseNPs ontaining hotel and the example WH-ngrams ontaining where. Note that the table doesn t inlude all the topi term andidates ontaining hotel or where. 84

5 WWW 2008 / Refereed Trak: Data Mining - Learning April 2-25, 2008 Beijing, China Redution of Topi Terms The redution of topi terms is needed in order to represent the extrated topi terms more ompatly as well as to judge the (degree of) reusability of the topi terms when applied to unseen data. That is to say, the redution is to generate a voabulary of topi terms whih well models both the given questions and the new queried question. For example, given the topi term andidates ontaining hotel as list in Table 2, we wish to redue the topi term embassy suite hotel as suite hotel beause embassy suite hotel is too sparse and unlikely to be hit by the new question posted by other users. At the same time, we wish to keep inexpensive western hotel although western hotel is also one of the topi terms. Formally, the redution of topi terms is a deision-making proess of, given a orpus of questions, what topi terms are more likely appliable to unseen questions. By the language of model seletion, it is to selet a model best fitting the given orpus and having good apability of generality. Within the model seletion, eah operation of redution of topi terms results in a different model. We make use of the MDL-based tree ut model for the model seletion. In the following, we are to explain how a tree of topi terms is onstruted suh that it an model the proess of reduing topi terms. As for redution, intuitively we want to ignore the modifier part of a topi term when reduing the topi term to another topi term. Thus, we define two types of redution: a) removing the prefixes of BaseNPs; b) removing the suffixes of WH-ngrams. We make use of the data struture of prefix tree (also known as trie) [0] for the representation of the BaseNPs or WH-ngrams. The two types of redution orrespond to two types of prefix tree, namely prefix tree of reversely-ordered BaseNPs and prefix tree of WH-ngrams. As for the reversely-ordered BaseNP, for example, beahfront hotel is rewritten as hotel beahfront. Figure 3 provides a prefix tree of the BaseNPs list in Table 2. Note that the orders of words are reversed before the BaseNPs are fed into the prefix tree. The numbers in the parentheses are the frequenies of the orresponding topi terms. For example, the node denoted by beahfront (5) means that the frequeny of beahfront hotel is 5, whih doesn t inlude the frequeny of good beahfront hotel and that of great beahfront hotel. Figure 4 provides a prefix tree of the WH-ngrams list in Table 2. Note that the funtional words suh as to and for are skipped when the WH-ngrams are fed into the prefix tree. For the prefix trees depited in Figure 3 and 4, the root node (required by the definition of prefix tree) whih is assoiated with empty string is ignored. Given a prefix tree of topi terms, we an then employ the MDL priniple for seleting the best ut. A prefix tree an have various uts whih orrespond to different hoies of topi terms. In Figure 3, the dot line and the dash line are just two of all the possible uts. The seletion given by the MDL is the ut indiated by the dash line, whih results in the new tree at the bottom of Figure 3. In the new tree, for example, the frequeny of beahfront hotel is updated as to inlude good beahfront hotel and great beahfront hotel. Similarly, the MDL an also give the best ut indiated by the dash line in Figure 4. Figure 3. The prefix tree of the BaseNPs about hotel Figure 4. The prefix tree of the WH-ngrams about where Linking Topi Terms Reall that our approah to question reommendation relies on a tree struture alled question tree whih onsists of all the topi terms ourring in either the queried question or the related questions. In this subsetion, with a series of definitions, we are to desribe how a question tree is onstruted from a olletion of questions. Definition (Topi Profile) The topi profile θ t of a topi term t in a ategorized text olletion is a probability distribution of ategories { p( t)} where C is a set of ategory. C where (365) buy (5) learn (6) tea (2) insurane (23) ginseng () omputer () tea (2) insurane (23) hotel (3983) nie (224) suite (3) western (40) embassy () nie (2) good (4) inexpensive (2) hotel (3983) where (3746) buy (6) learn (8) ount(, t) p( t) = (7) ount(, t) C where ount (, t) is the frequeny of the topi term t within the ategory. Clearly, we have p ( t) =. C beahfront (5) affordable (248) good (3) great (3) nie (224) suite (6) western (40) beahfront () affordable (248) good (4) inexpensive (2) Japanese () By ategorized questions, we refer to the questions that are organized in a tree of taxonomy. For example, at Yahoo! Answers, the question How do I install my wireless router is ategorized as Computers & Internet Computer Networking. 85

6 WWW 2008 / Refereed Trak: Data Mining - Learning April 2-25, 2008 Beijing, China Definition 2 (Speifiity) The speifiity s (t) of a topi term t is the inverse of the entropy of the topi profile θ t. More speifially, s( t) = ( p( t)log p( t) + ε ) (8) C where ε is a smoothing parameter used to ope with the topi terms whose entropy are 0. In our pratie, the value of ε is set as We use speifiity to represent how speifi a topi term is in haraterizing information needs of users who post questions. A topi term of high speifiity (e.g., Hamburg, Berlin) usually speifies the question topi orresponding to the main ontext of a question. Thus, a good question reommendation is required to keep it as muh as possible so that the reommendation an be around the same ontext. A topi term of low speifiity is usually used to represent the question fous (e.g., ool lub, where to see) whih is relatively volatile. Atually, it is natural to think about an alternative definition of speifiity. That is, defining speifiity of a topi term t as the inverse of the number of ategories in whih t is mentioned. s '( t) = δ ( ount(, t) > 0) C where δ ( ) is an indiator funtion suh that it equals to when the ondition is satisfied and 0 otherwise. Let s see why the alternative definition is not appropriate for weighing topi terms. For example, in the ategory Europe of Yahoo! Answers, you may find a question I am from Beijing and like to have your reommendation on where to see in Berlin. The similar questions ontaining Beijing an also be found in other ategories although most ourrenes of Beijing are in the ategory Asia Paifi China. The alternative definition of speifiity may mistakenly onsider Beijing as a very general topi term. Definition 3 (Topi Chain) A topi hain q of a question q is a sequene of ordered topi terms t t 2 t m suh that ) t is inluded in i q, i m ; 2) s ( tk ) > s( tl ), k < l m. For example, the topi hain of any ool lubs in Berlin or Hamburg? is Hamburg Berlin ool lub beause the speifiities for Hamburg, Berlin, and ool lub are 0.99, 0.62, and Definition 4 (Question Tree) A question tree of a question set N Q = { q i } is a prefix tree built over the topi hains N i= Q = { qi } i= of the question set Q. Clearly, if a question set ontains only one question, its question tree will be exatly same as the topi hain of the question. Note that the root node of a question tree is assoiated with empty string as the definition of prefix tree requires [0]. Given the following topi hains with respet to the questions in Figure, Hamburg Berlin ool lub Hamburg Berlin where to see Hamburg Berlin how far Hamburg Berlin how long does it take Hamburg heap hotel we an have the question tree presented in the middle of Figure. 4.3 Ranking Reommendation Candidates As introdued in Setion 3, question reommendation defined in this paper is onduted by substituting onsistent topi terms. Therefore, given a question represented as a topi hain, we should be able to ) determine what onsistent topi terms (representing question fous) should be substituted, and 2) sore the reommendation andidates rendered by various substitutions. We will address these two issues in the following sub-setions Determination of Substitutions The topi terms of low speifiity are usually used to represent the question fous (e.g., ool lub, where to see), whih are relatively volatile. Determination of substitutions is to disriminate these topi terms from those of high speifiity and then suggest them as substitutions. Reall that the topi terms in a topi hain of question are ordered aording to their speifiity values. Thus, a ut of a topi hain naturally gives a deision of disriminating the topi terms of low speifiity (representing question fous) from the topi terms of high speifiity (representing question topi). Given a topi hain of question onsisting of m topi terms, there exist ( m ) possible uts. Eah possible ut gives one kind of suggestion on substitution. A straightforward method is just to take the ( m ) uts and then on the basis of them suggest ( m ) kinds of substitutions. However, suh a simple method ompliates the problem of ranking reommendation andidates for it introdues too muh unertainty. In addition to that, the simple method annot leverage the related questions whih provide more observations of the topi terms. Thus, we propose using the MDL-based tree ut model for the searh of best ut of a topi hain. The best ut will be unique for a single topi hain. Given a topi hain q of a question q, we are to onstrut a question tree of the related questions as follow: ) Collet a set of topi hains N Q = { qi } suh that at least i= one topi term ours in both q and q i 2) Construt a question tree from the set of topi hains Q Uq Then, with the MDL-based tree ut model, we an obtain a best ut of the question tree, whih also gives a ut for eah topi hain (inluding q ). The uniqueness of the best ut an dramatially redue the unertainty of reommendation ranking, ompared to the simple method. In addition to that, the best ut is obtained by observing the distribution of topi terms over all the potential reommendations (all the questions related to a queried question), instead of the queried question only. A ut of a topi hain q separates the topi hain in two parts: HEAD and TAIL. HEAD (denoted as H ( q ) is the subsequene of the original topi hain q before the ut. TAIL (denoted as 86

7 WWW 2008 / Refereed Trak: Data Mining - Learning T ( q ) is the subsequene of the original topi hain q after the ut. Thus, q = H ( q ) T ( q ) Sores of Reommendation Candidates The ranking of reommendation andidates an be based on a reommendation sore r ( q ~ q) defined over a queried question q and a reommendation andidate q ~ suh that, given that ~q and ~q are both reommendation andidates of the query 2 q, ~q is the better reommendation of q than ~q if 2 r ( q~ ) ( ~ q > r q2 q). Given that the topi hain of a queried question q is separated as q = H ( q ) T ( q ) by a ut and the topi hain of a reommendation andidate q ~ is separated as q ~ = H ( q ~ ) T ( q ~ ), we further require that the reommendation sore r ( q ~ q) satisfies, Speifiity: The more similar are H ( q ) and H ( q ~ ), the greater r ( q ~ q) is; Generality: The more similar are T ( q ) and T ( q ~ ), the less r ( q~ q) is. The requirements assure that the substitutions given by reommendation andidates fous on the TAIL part of the topi hain, whih provides users the opportunity of exploring different question fous (e.g., ool lub vs. where to see) around the same question topi (e.g., Hamburg, Berlin). In order to define the reommendation sore, we first introdue a sore sim q q ) measuring the similarity of the topi hain to q 2, ( 2 sim q2 ) (9) ( q ) = s( t PMI t t ) max (, t2 q 2 2 q t q where q represents the number of topi terms ontained in q. PMI ( t, t 2) represents the PMI (Pointwise Mutual Information) [6] of a pair of topi terms t and t 2. In our experiments, the PMI values are alulated over questions. Aording to the equation, the similarity between topi hains is basially determined by the assoiations between onsistent topi terms. The PMI values of individual pair of topi terms in the equation are weighed by the speifiity of topi terms ourring in q. Note that the similarity defined here is asymmetri. Then, we define the reommendation sore r ( q ~ q) as, ( ~ ( ~ ) ( ) ( )) ( ) ( ( ~ r q q = λ sim H q H q λ sim T q ) T( q )) (0) This equation balanes between the two requirements of speifiity and generality in a way of linear interpolation. The higher value of λ implies that the reommendations tend to be similar to the queried question. While, the lower value of λ enourages the reommendations to explore the question fous different from that of the queried questions. 5. EXPERIMENTAL RESULTS We have onduted experiments to verify the effetiveness of our approah to question reommendation. Partiularly, we have investigated the use of the MDL-based tree ut model. q 5. Dataset and Evaluation Measures Dataset We made use of the questions rawled from Yahoo! Answers for the evaluation. More speifially, we utilize the resolved questions under two of the top-level ategories at Yahoo! Answers, namely travel and omputers & internet. The rawled questions inlude 34,66 items from the travel ategory and 20,785 items from the omputers & internet ategory. Eah resolved question onsists of three fields: title, desription, and answers. We developed two test sets, one for the ategory travel denoted as TREVAL TST, and the other for omputers & internet denoted as COM-INT TST. In order to reate the test sets, we randomly seleted 00 questions for eah ategory. To obtain good question reommendations given a queried question, we employed the Vetor Spae Model (VSM) [22] to retrieve the top 20 results and did manual judgments. Given a returned result by VSM, an assessor is asked to label it with relevant or irrelevant. If a returned result is onsidered as a reommendation for the queried question, the assessor will label it relevant ; otherwise, the assessor will label it as irrelevant. Two assessors were involved in the manual judgments. Eah of them was asked to label 50 questions from TRAVEL TST and 50 from COM-INT TST. In the proess of manual judgment, the assessors were presented only the titles of the questions (for both the queried questions and the returned questions). We assume that the titles of the questions already provide enough ontextual information for understanding users information needs. Table 3 provides the statistis on the final test set. Table 3. Statistis on the test data # Queries # Returned # Relevant TRAVEL TST 00 2, COM-INT TST 00 2, Baseline and Evaluation Measures We utilized two baseline methods for demonstrating the effetiveness of our approah, the word-based VSM and the phrase-based VSM. The word-based VSM is just the onventional VSM [22] for information retrieval in whih words are used to represent queries and douments (in our ase, they are questions). The phrase-based VSM is similar to the word-based VSM but it uses the extrated topi terms to represent questions. We made use of three measures for evaluating the results of question reommendation methods. They are MAP, R-preision, and Top N preision (P@N). MAP is widely used in IR and is based on the assumption that there are two ategories: positive (relevant) and negative (irrelevant) in ranking of instanes (questions). MAP alulates the mean of average preisions over a set of queries. Given a query q i, its average preision (AvgP i ) is defined as the average of preision after eah positive (relevant) instane is retrieved. M P( j) pos( j) AvgPi = () number of positive instanes j= April 2-25, 2008 Beijing, China where j is the rank, M is the number of instanes retrieved, pos(j) is a binary funtion to indiate whether the instane in the rank j is positive (relevant), and P(j) is the preision at the given ut-off rank j: 87

8 WWW 2008 / Refereed Trak: Data Mining - Learning April 2-25, 2008 Beijing, China number of positive instanes in top j positions P( j) = (2) j R-preision is defined as K P( Ri ) i= (3) R preision = K where R i is number of reommendations (labeled as relevant ) for the query i in the ground truth. K is the total number of queries (topis) in the evaluation set. Top N preision (P@N) is defined as N Here, N =, 2,..,0. K i = = P ( N ) K (4) 5.2 Reommending Questions about Travel In the experiments, we made use of the questions about travel to test the performane of the proposed MDL-based approah to question reommendation. More speifially, we used the 00 queries in the test set TRAVEL TST to searh for reommendations from the 34,66 questions ategorized as travel. Note that only the questions ourring in the test set an be evaluated. We made use of the taxonomy of questions provided at Yahoo! Answers for the alulation of speifiity of topi terms. The taxonomy is organized in a tree struture. In the following experiments, we only utilized as the ategories of questions the leaf nodes of the taxonomy tree (regarding travel ), whih inludes 355 ategories. We utilized the question titles only for the extration of topi term and the alulation of PMI (Pointwise Mutual Information) [6]. We randomly divided the test queries into five even subsets and onduted 5-fold ross-validation experiments. In eah trial, we tuned the parameter λ in the equation (0) with four of the five subsets and then applied it to one remaining subset. The experimental results reported below (exept that in Figure 5) are those averaged over the five trials. Basi results: In Table 4, our approah inludes all the omponents based on the MDL suh as the MDL-based redution of topi terms and the MDL-based seletion of substitution. Table 4. Reommending questions about travel Methods R-Preision P@5 MAP VSM PVSM Our approah From Table 4, we see that our approah outperforms the baseline approahes VSM and PVSM in terms of all the measures. We onduted signifiant test (t-test) on the improvements of our approah over VSM and PVSM in terms of R-Preision and P@5. The result indiates that the improvements are statistially signifiant (p-value < 0.05). We also onduted t-test on the improvements of our approah over VSM and PVSM in terms of MAP. The result indiates that the improvements are not statistially signifiant. That is to say, ompared to the baseline methods, our approah does well on the top-rank results while ahieving omparable performane on all the returned results. Note that the average number of the relevant results for a queried question about travel is 4.05 (405/00), whih means that the R-Preision is an approximation of P@4. Table 5 provides the TOP-3 reommendations whih are given by VSM, PVSM, and our approah respetively. Both VSM and PVSM return as the top- reommendation the question whih is semantially equivalent to the queried question. The top-2/top-3 reommendations given by VSM and the top-2 given by PVSM provide the hotel information around the loations other than downtown Chiago. Atually, all these reommendations annot help the user (who posted the queried question) explore other aspets of users interest ( downtown Chiago ). The reason is that neither VSM nor PVSM is aware that the question topi of the queried question is the downtown Chiago. In ontrast, our approah an provide other aspets (question fous) about downtown Chiago beause of its awareness of the question topi. Table 5. Reommendations for What's a good but heap hotel/motel/anything in downtown Chiago? Methods Reommendations VSM. What is a lean, heap hotel near the downtown Chiago? 2. What is a good heap hotel/motel near Disneyland? 3. What is a good heap motel in Tapei? PVSM. What is a lean, heap hotel near the downtown Chiago? 2. Is there any heap hotel/motel in Calgary Alberta? 3. What is there to do in downtown Chiago? Our approah. What is there to do in downtown Chiago? 2. What are some fun heap/free things to do & see in downtown Chiago? 3. What's the ost of a ab in downtown Chiago? Effetiveness of MDL: The major advantage of our approah is that the MDL-based tree ut model enables itself to leverage the global ontext for both redution of topi terms and seletion of substitution. In redution of topi terms, the deision of reduing a topi term or not is made by observing the empirial distribution of all the extrated topi terms over the entire data olletion. In seletion of substitution, the deision of substituting a topi term or not is made by observing the entire related questions as well as the queried question. To see how muh the MDL-based tree ut model benefit the question reommendation, we introdue another three baseline methods for omparison. The first method (denoted as First ) is made by removing the omponent redution of topi terms. The seond (denoted as Seond ) is to replae the MDL-based seletion of substitution with the simple method of onsidering all the possible uts of topi hains. In the simple method, any link within the topi hain of the queried question is onsidered as a ut. The third method (denoted as Third ), both removing the redution of topi terms and replaing the MDL-based seletion of substitution, is the ombination of the first and the seond. 88

9 WWW 2008 / Refereed Trak: Data Mining - Learning April 2-25, 2008 Beijing, China Table 6. The advantage of the MDL-based tree ut model Methods R-Preision P@5 MAP First Seond Third Our approah Table 6 provides the omparison. From Table 6, we see that either removing redution of topi terms or replaing the MDL-based seletion of substitution impairs the performane of question reommendation. The t-test also onfirms that the performane drop rendered by replaing the MDL-based seletion of substitution is statistially signifiant (p-value < 0.05). Although the improvement ontributed by redution of topi terms is not statistially signifiant, we still argue its use. This is beause the improvement was ahieved while the size of the voabulary of topi terms was dereased dramatially. The size of the voabulary is 289,25 before the redution and 73,202 after the redution. The redution in size is about 40%. The Use of Different Types of Topi Term In this experiment, we are to see the use of the two types of topi term, namely BaseNP and WH-ngram. Table 7 illustrates that both BaseNPs and WH-ngrams are useful for haraterizing the questions. In Table 7, the method - BaseNP denotes that of solely using WH-ngams in extration of topi terms and - WHngram denotes that of solely using BaseNPs in extration of topi terms. To see what roles the BaseNPs and WH-ngrams play in the topi hains, we further analyzed the 00 test queries (not inluding the reommendations) whose best uts were given by the MDL-based tree ut model. The analysis results show that 63% (89/30) noun phrases our in the HEAD of topi hains and 4% (25/6) WH-ngrams our in the HEAD part. In other words, WH-ngrams are used as the question fous more often than noun phrases. Table 7. The use of noun phrases and WH-ngrams Methods R-Preision P@5 MAP - BaseNP WH-ngram Our approah Aggregation Strategy: The equation (9) provides one strategy, max, for aggregating the individual assoiation of topi terms from two topi hains. Alternatively we an also make use of another strategy, average, as given by the equation (5). sim( q (5) 2 q ) = s( t ) PMI( t, t2) q q 2 t q, t2 q2 With max strategy, the similarity of two topi hains is determined by the similarity of the pair of topi terms whih are most lose to eah other in terms of PMI. With the average strategy, the similarity of two topi hains is determined by the average of similarities of the pairs of topi terms. Table 8. Aggregation strategy: max vs. average Methods R-Preision MAP P@5 average max From Table 8, we see that the max strategy outperforms the average strategy signifiantly (p-value < 0.05). This suggests that the similarity of two topi hains is better basing on the similarity of the losest topi terms (in terms of PMI) from the two topi hains. Balane between Speifiity and Generality In equation (0), we use the parameter λ to trade-off the speifiity and generality of reommendations. The higher value of implies that the reommendations tend to be similar to the queried question. While, the lower value of λ biases the reommendations to explore the question fous different from that of the queried questions. In this experiment, we are to explore how influential the value of λ is on the performane of question reommendation. Figure 5 provides the hange of R-Preision with respet to λ. The result was obtained with the 00 queries diretly, instead of the five-fold ross-validation. From Figure 5, we see that the proposed approah performs best when λ is around 0.7. Therefore, at the point of 0.7, the question reommendation an best balane the speifiity and generality. Figure 5. Balaning between speifiity and generality. However, one might expet that the best λ is around 0.5. The reason for the existene of the ontroversy (0.7 vs. 0.5) is that the similarity between two topi terms in the HEAD of a topi hain is not of the same sale as that between topi terms in the TAIL. For example, it is unusual to see the o-ourrene of where to go and where to see within a single question, whih implies almost 00% negative assoiation. In ontrast, we an observe the o-ourrene of Hamburg and Berlin within a single question, whih gives a moderate positive assoiation. Thus, the question reommendation tends to use a bigger value of λ for the speifiity (the HEAD part of a topi hain) as a re-saling mehanism as well. 5.3 Reommending Questions about Computers & Internet In the experiments, we made use of the questions about omputers & internet to test the performane of the proposed MDL-based approah to question reommendation. More speifially, we used the 00 queries in the test set COM-INT TST to searh for reommendations from the 20,785 questions ategorized as omputers & internet. We utilized as the ategories of questions the leaf nodes of the taxonomy tree regarding omputers & Internet, whih inlude 23 ategories. We made use of only the question titles for the extration of topi term and the alulation of PMI. Again we onduted 5-fold ross-validation for the tuning of the parameter λ. The experimental results reported in Table 9 are averaged over the five trials. In Table 9, our approah made use of the same tehnique as presented in Table 4. 89

10 WWW 2008 / Refereed Trak: Data Mining - Learning April 2-25, 2008 Beijing, China Table 9. Reommending questions about omputers & Internet Methods R-Preision P@5 MAP VSM PVSM Our approah Again, we see that our approah outperforms the baseline approahes VSM and PVSM in terms of all the measures. We onduted signifiant test (t-test) on the improvements of our approah over VSM and PVSM. The result indiates that the improvements in terms of R-Preision and P@5 are statistially signifiant (p-value < 0.05). Thus, we an draw the same onlusion with the omputers & internet data as that with the travel data. That is, ompared to the baseline methods, our approah does well on the top-rank results while ahieving omparable performane on all the returned results. Note that the average number of the relevant results for a queried question about omputers & internet is 3.86 (386/00), whih means that the R-Preision is an approximation of P@4. 6. CONCLUSIONS In this paper, we have proposed to address the problem of question reommendation as a omplement of question searh. The question reommendation was defined as substituting question fous. Under the setting, we have developed a new approah whih onsists of two steps: representing questions as trees (graphs) of topi terms and ranking reommendation andidates. We have proposed using the MDL-based tree ut model for takling the issues involved in these two steps. Experimental results indiate that our approah performs signifiantly better than the baseline methods of the word-based VSM and the phrase-based VSM. The results also show that the use of the MDL-based tree ut model is essential to our approah. Though only utilizing the data from ommunity-based question answering servie (QA), we also an find the ategorized questions at forum sites or FAQ sites. Thus, as one of our future work, we will try to investigate the effetiveness of the proposed approah for other kinds of web servies. The proposed approah is not limited to question reommendation. Long queries of web searh usually provide enough ontext information suh as topi and fous as questions do. Thus, as the other future work, we will try to apply the proposed approah to query suggestion of long queries. ACKNOWLEDGMENTS We thank the anonymous reviewers for their valuable omments and suggestions to this paper. REFERENCES [] Banerjee, S. and Pedersen, T. The design, implementation, and use of the ngram statistis pakage. In Pro. of the 4 th CICLing'03. [2] Barron, A., Rissanen, J., and Yu, B. The minimum desription length priniple in oding and modeling. IEEE Trans. Information Theory, vol. 44 (998), pp [3] Burke, R. D., Hammond, K. J., Kulyukin, V. A., Lytinen, S. L., Tomuro, N., and Shoenberg, S. Question answering from frequently asked question files: Experienes with the faq finder system. Tehnial report, 997. [4] Cao, Y. and Li, H. Base noun phrase translation using web data and the EM algorithm. In Pro. of COLING'02. [5] Christopher, M. D. and Hinrih, S. Foundations of Statistial Natural Language Proessing. MIT Press: 999. [6] Churh, K. W. and Hanks, P. Word assoiation norms, mutual information, and lexiography. In Pro. of ACL'89. [7] Cuerzan, S. and White, R. W. Query Suggestion based on landing pages. In Pro. of SIGIR'07. [8] Fellbaum, C. WordNet: An Eletroni Lexial Database. MIT Press, 998. [9] Fonsea, B. M., Golgher, P. B., Moura, E. S., Possas, B., and Ziviani, N. Disovering searh engine related queries using assoiation rules. Journal of Web Engineering, [0] Fredkin, E. Trie Memory. Communiations of the ACM, D. 3(9): , 960. [] Gleih, D. and Zhukov, L. SVD based term suggestion and ranking system. In Pro. of ICDM'04. [2] Jensen, E. C., Beitzel, S. M., Chowdhury, A., and Frieder, O. Query phrase suggestion from topially tagged session logs. In Pro. of FQAS'06. [3] Jeon, J. and Croft, W. B. Learning translation-based language models using Q&A arhives. Tehnial Report, University of Massahusetts. [4] Jeon, J., Croft, W. B., and Lee, J. Finding similar questions in large question and answer arhives. In Pro. of CIKM'05. [5] Jeon, J., Croft, W. B., and Lee, J. H. Finding semantially similar questions based on their answers. In Pro. of SIGIR'05. [6] Jones, R., Rey, B., Madani, O., and Greiner, W. Generating query substitutions. In Pro. of WWW'06. [7] Kawamae, N., Suzuki, H., and Mizuno, O. Query and ontent suggestion based on latent interest and topi lass. In Pro. of WWW'04. [8] Lai, Y.-S., Fung, K.-A., and Wu, C.-H. Faq mining via list detetion. In Pro. of the Workshop on Multilingual Summarization and Question Answering, [9] Li, H. and Abe, N. Generalizing Case Frames using a thesaurus and the MDL priniple. Computational Linguistis, 24(2), pp , 998. [20] Rissanen, J. Modeling by shortest data desription. Automatia, vol. 4 (978), pp [2] Rissanen, J. Universal oding information, predition and estimation. IEEE Transation on Information Theory, vol. 30(4): [22] Salton, G., Wong, A., and Yang, C. S. A vetor spae model for automati indexing. Communiations of the ACM, vol. 8, nr., pages [23] Sneiders, E. Automated question answering using question templates that over the oneptual model of the database. In Pro. of the 6 th NLDB'02. [24] Wen, J. R., Nie, J.-Y., and Zhang, H. J. Query lustering using user logs. ACM Trans. Information Systems, 20():59-8, [25] Xun, E. D., Huang, C.N., and Zhou, M. A unified statistial model for the identifiation of English BaseNP. In Pro. of ACL'00. 90