A Simple and Efficient Sampling Method for Estimating AP and NDCG

Size: px
Start display at page:

Download "A Simple and Efficient Sampling Method for Estimating AP and NDCG"

Transcription

1 A Simple and Efficient Sampling Method for Etimating AP and NDCG Emine Yilmaz Microoft Reearch 7 JJ Thomon Avenue Cambridge CB3 FB, UK Evangelo Kanoula ekanou@cc.neu.edu Javed A. Alam jaa@cc.neu.edu College of Computer and Information Science Northeatern Univerity 36 Huntington Ave, #22 WVH Boton, MA 25 ABSTRACT We conider the problem of large cale retrieval evaluation. Recently two method baed on random ampling were propoed a a olution to the extenive effort required to judge ten of thouand of document. While the firt method propoed by Alam et al. [] i quite accurate and efficient, it i overly complex, making it difficult to be ued by the community, and while the econd method propoed by Yilmaz et al., [4], i relatively imple, it i le efficient than the former ince it employ uniform random ampling from the et of complete judgment. Further, none of thee method provide confidence interval on the etimated value. The contribution of thi paper i threefold: () we derive confidence interval for, (2) we extend to incorporate nonrandom relevance judgment by employing tratified random ampling, hence combining the efficiency of tratification with the implicity of random ampling, (3) we decribe how thi approach can be utilized to etimate ndcg from incomplete judgment. We validate the propoed method uing TREC data and demontrate that thee new method can be ued to incorporate nonrandom ample, a were available in TREC Terabyte track 6. Categorie and Subject Decriptor: H.3 Information Storage and Retrieval; H.3.4 Sytem and Software: Performance Evaluation General Term: Experimentation, Meaurement, Theory Keyword: Evaluation, Sampling, Incomplete Judgment, Average Preciion, ndcg,. INTRODUCTION We conider the problem of large cale retrieval evaluation, in particular, retrieval evaluation with incomplete relevance judgment. We gratefully acknowledge the upport provided by NSF grant IIS and IIS Permiion to make digital or hard copie of all or part of thi work for peronal or claroom ue i granted without fee provided that copie are not made or ditributed for profit or commercial advantage and that copie bear thi notice and the full citation on the firt page. To copy otherwie, to republih, to pot on erver or to reditribute to lit, require prior pecific permiion and/or a fee. SIGIR 8, July 2 24, 28, Singapore. Copyright 28 ACM /8/7...$5.. The mot commonly employed methodology for aeing the quality retrieval ytem i the Cranfield methodology adopted by TREC [7]. The main aumption behind thi methodology i that the relevance judgment are complete, i.e., for each query, all document in the collection relevant to thi query are identified. When the document collection i large, obtaining complete relevance judgment i infeaible due to the need for extenive human effort. Intead, TREC employ depth-k pooling (typically, depth- pooling) to overcome thi burden, combining top k document retrieved by the ubmitted ytem and auming the ret of the document are nonrelevant. While a depth- pool i ignificantly maller than the document collection, it till require extenive judgment effort (e.g. 86,83 judgment for ). Furthermore, even though depth- pool were hown to contain mot of the relevant document [8], the ize of document collection tend to increae and thu thee pool may be inadequate for identifying mot of the relevant document. Recently, evaluation with incomplete relevance judgment ha gained attention a a olution to the problem of extenive judgment effort. New evaluation meaure have been propoed in the literature [, 5,, 2, 4] ince tandard evaluation meaure uch a average preciion are not robut to incomplete judgment [3]. A a olution to thi problem, Buckley and Voorhee [3] propoed bpref, a now commonly ued meaure by information retrieval community. Sakai [2] intead applied traditional meaure to condened lit of document obtained by filtering out all unjudged document from the original ranked lit and howed that thee verion of meaure are actually more robut to incompletene than bpref. Carterette et al. [5] and Moffat et al. [] elect a ubet of document to be judged baed on the benefit document provide in fully ranking ytem or identifying the bet ytem, repectively. Even though the aforementioned approache are all hown to be more robut to incompletene than tandard evaluation meaure, thee method are not guaranteed to compute or etimate the value of tandard evaluation meaure. Hence, the value of meaure obtained by thee method are difficult to interpret. Yilmaz and Alam [4] and Alam et al. [] intead ue random ampling to etimate the actual value of average preciion when relevance judgment are incomplete. Both of thee method are baed on treating incomplete relevance

2 judgment a a ample drawn from the et of complete judgment and uing tatitical method to etimate the actual value of the meaure. Thee method are both hown to () produce unbiaed etimate of average preciion even when relevance judgment are incomplete and (2) be more robut to incomplete relevance judgment than any other meaure uch a bpref [3] or the condened verion of the meaure (referred to a induced AP in the paper)[4]. The meaure propoed by Yilmaz and Alam, [4], became a commonly ued meaure by information retrieval community [2, 3] and wa ued in TREC VID and Terabyte track in 26 [, 4]. A limitation of accrue from the meaure aumption that incomplete relevance judgment are a imple random ample drawn from the et of complete judgment. Typical evaluation meaure give more weight to document retrieved toward the top of a retrieved lit and therefore, a top-heavy ampling trategy would lead to more accurate reult with higher efficiency in term of judgment effort needed. On the other hand, according to the method by Alam et al. [] ample are drawn according to a carefully choen non-uniform ditribution over the document in the depth- pool. Even though thi method i more efficient in term of judgment effort than, it i very complex both in conception and implementation and therefore le applicable. Furthermore, although average preciion etimator a propoed by both of the aforementioned method are unbiaed in expectation, in practice, when calculated uing a ingle ample of relevance judgment, may vary in value. Thi neceitate the derivation and ue of confidence interval around the etimated value in order to allow confident concluion regarding the actual value of average preciion and thu the ranking of retrieval ytem. In thi paper, we mainly focu on inferred average preciion. Firt, we derive confidence interval for the meaure and validate them uing TREC data. We how that along with the correponding confidence interval can allow reearcher to reach confident concluion about actual average preciion, even when relevance judgment are incomplete. We then focu on the efficiency of the meaure. We employ a tratified random ampling methodology and extend the meaure to incorporate relevance judgment created according to any uch ampling ditribution. Thi extended combine the implicity of random ampling with the efficiency of tratification and thu it i imple and eay to compute while, at the ame time, it i much more efficient than in term of reducing the judgment effort. We further claim that the ame methodology can be applied to other evaluation meaure and demontrate how ndcg (a commonly ued meaure that incorporate graded relevance judgment [9]) can be etimated uing incomplete relevance judgment. 2. CONFIDENCE INTERVALS FOR INFAP The inferred average preciion, by tatitical contruction, i an unbiaed etimator of average preciion and thu it i deigned to be exactly equal to average preciion in expectation. However in practice, it may be low or high due to the nature of ampling (epecially when the ubet of document whoe binary relevance i available i mall). In other word, there i variability in the value of becaue different ample from the collection of document give rie to different value of. The amount of the variability in i meaured by it variance. Before computing the variance of let reviit the random experiment whoe expectation i average preciion [4] and identify all ource of variability in the outcome of thi random experiment. Given a ranked lit of document with repect to a given topic:. Select a relevant document at random and let the rank of thi relevant document in lit be k. 2. Select a rank, j, at random from the et {,..., k}. 3. Output the binary relevance of the document at rank j. In expectation, tep (2) and (3) effectively compute the preciion at a relevant document and in combination, tep () compute the average of thee preciion. The aforementioned experiment can be realized a a twotage ampling. At the firt tage tep () a ample of cut-off level at relevant document i elected. The value i computed a an average of the etimated preciion value at the ampled cut-off level. Even if we aume that thee preciion value are the actual preciion value, varie becaue different ample of cut-off level will reult in different value of. Therefore, computing uing preciion value only at a ubet of cut-off level introduce the firt component of variability. Let rel be the et of the judged relevant document of ize r. Thi firt variance component can be etimated a var. comp. = ( p) 2 /r where p % i the ampling percentage and 2 the variance among the preciion value at the judged relevant document P calculated a 2 = k rel (d P C k ) /r. At the econd tage tep (2) for each one of the elected cut-off level, a ample of document above that cutoff level document i ued to etimate the correponding to the cut-off preciion value. Therefore, even for a given ample of cut-off level, ha variability becaue different ample of document give rie to different value of preciion and thu different value of. Hence, computing the preciion at ome cut-off uing only a ubet of the document above that cut-off introduce a econd component of variability. Auming that preciion at different cut-off level are independent from each other, thi econd variance component can be etimated a, X var. comp. 2 = var[ d P C k ] /r 2 k rel where var[ P d C k ] i the variance of the etimated preciion at cut-off k. According to the Law of Total Variance, the total variance of can be computed a the um of the two aforementioned variance component; hence, var[] = ( p) 2 r + P k rel var[d P C k ] r 2 The complete formula of variance along with the derivation can be found at the Appendix

3 When evaluating retrieval ytem, the average of value acro all topic (mean ) i employed. The variance of the mean can be computed a a function of the variance of a var[mean ] = X var[]/(# of querie) 2 According to the Central Limit Theorem one can aign 95% confidence interval to mean a a function of it variance. A 95% confidence interval centered at the mean intimate that with 95% confidence the actual value of MAP i within thi interval. We ued,9 and data to validate the derived variance of the mean when relevance judgment are incomplete. We imulated the effect of incomplete relevance judgment a in [4]. For each TREC, we formed incomplete judgment et by ampling from the entire depth- pool over all ubmitted run. Thi i done by electing p% of the complete judgment et uniformly at random, where p {, 2, 3}. The reult of our experiment led to identical concluion over all TREC dataet and therefore, due to pace limitation, we report only reult for. Figure illutrate the mean value computed from a ingle random ample of document per topic for each run againt the actual MAP value for p {, 2, 3} for. The 95% confidence interval are depicted a error bar around the mean value. A one can oberve, the greatet majority of the confidence interval interect the 45 o dahed line indicating that the greatet majority of the confidence interval cover the actual MAP value. Furthermore, we computed the mean value and the correponding confidence interval for different ampling trial over data and we accumulated the deviation of the computed mean value from the actual MAP value in term of tandard deviation. Thi way we generated a Cumulative Ditribution Function of divergence of mean value per ytem. According to the Central Limit Theorem each of thee CDF hould match the CDF of the Normal Ditribution. We performed a Kolmogorov- Smirnov tet of fitne and for 9% of the ytem the hypothei that the two CDF match could not be rejected (α =.5) which validate our derived theoretical reult. 3. INFERRED AP ON NONRANDOM JUDGMENTS In the previou ection we derived confidence interval for in a etup where document to be judged were a random ubet of the entire document collection. Confidence interval can be further reduced (i.e. the accuracy of the etimator can be improved) by utilizing a top-heavy ampling trategy. In thi ection we conider a etup where relevance judgment are not a random ubet of complete judgment and how how can be extended to produce unbiaed etimate of average preciion in uch a etup. We denote the extended meaure a x. Similar to the paradigm, conider the cae where we would like to evaluate the quality of retrieval ytem with repect to a complete pool and aume that relevant judgment are incomplete. Further aume that the et of available judgment are contructed by diving the complete collection of document into dijoint contiguou ubet (trata) and then randomly electing (ampling) ome document from each tratum to be judged. The ampling within each tratum i performed independently, therefore, the ampling percentage can be choen to be different for each tratum. For intance, one could chooe to plit the collection of document into two trata (baed on where they appear in the output of earch engine), and ample 9% of the document from the firt tratum and 3% of the document from the econd tratum. In effect, one could think a large variety of ampling trategie in term of thi multi-trata trategy. For example, the ampling trategy propoed by Alam et al. [] can be thought a each tratum containing a ingle document, with different ampling probabilitie aigned to different trata. Let AP d be the random variable correponding to the etimated average preciion of a ytem. Now conider the firt tep of the random experiment whoe expectation correpond to average preciion, i.e. picking a relevant document at random. Note that in the above etup, thi relevant document could fall into any one of the different trata. Since the et of document contained in the trata are dijoint, by definition of conditional expectation, one can write E[ AP d ] a: E[ AP d X ] = P E [ AP d ] Strata where P correpond to the probability of picking the relevant document from tratum and E [ AP d ] correpond to the expected value of average preciion given that the relevant document wa picked from tratum. Let R Q be the total number of relevant document in the complete judgment et and R be the total number of relevant document in tratum if we were to have all complete relevance judgment. Then, ince electing document from different trata i independent for each tratum, the probability of picking a relevant document from tratum i, P = R /R Q. Computing the actual value of R Q and R i not poible, ince the complete et of judgment i not available. However, we can etimate their value uing the incomplete relevance judgment. Let r be the number of ampled relevant document and n be the total number of ampled document from tratum. Furthermore, let N be the total number of document in tratum. Since the n document were ampled uniformly from tratum, the etimated number of relevant document within tratum, ˆR, can be computed a ˆR = (r /n ) N. Then the number of relevant document in query Q can be etimated a the um of thee etimate over all trata, i.e. ˆRQ = P ˆR. Given thee etimate, the probability of picking a relevant document from tratum can be etimated by, ˆP = ˆR / ˆR Q. Now, we need to compute the expected value of etimated average preciion, E [ AP d ], if we were to pick a relevant document at random from tratum. Since the incomplete relevance judgment within each tratum i a uniform random ubet of the judgment in that tratum, the induced ditribution over relevant document within each tratum i alo uniform, a deired. Therefore, the probability of picking any relevant document within thi tratum i equal. Hence, the expected etimated average preciion value within each tratum, E [ AP d ], can be computed a the average of the preciion at judged (ampled) relevant document within that tratum.

4 .6, Sample Percentage =%.6, Sample Percentage =2%, Sample Percentage =3% Inferred MAP.3.2 Inferred MAP.3.2 Inferred MAP Actual MAP Actual MAP Actual MAP Figure : TREC-8 mean inferred AP along with etimated confidence interval when relevance judgment are generated by ampling, 2 and 3 % of the depth- pool veru the mean actual AP. Now conider computing the expected preciion at a relevant document at rank k, which correpond to the expected outcome of picking a document at or above rank k and outputting the binary relevance of the document at thi rank (tep 2 and 3 of the random experiment). When picking a document at random at or above rank k and outputting the binary relevance of that document, one of the following two cae may occur. With probability /k, we pick the current document, and ince thi document i by definition relevant the outcome i. With probability (k )/k we pick a document above the current document, in which cae we need to calculate the expected preciion (or expected binary relevance) with repect to the document above rank k. Thu, E[ d P C k ] = k + k k E[d P C above k ] Let N k be the total number of document above rank k that belong tratum, n k be the total number of judged (ampled) document above rank k that belong to tratum and r k be the total number of judged (ampled) relevant document above rank k that alo belong to tratum. When computing the expected preciion within the (k ) document above rank k, with probability N k /(k ) we pick a document from tratum. Therefore, the expected preciion above rank k can be written a: E[prec above k] = X N k k E[d P C above k ] where E [ P d C above k ] i the expected preciion above rank k within tratum. Since we have a uniform ample of judged document from tratum, we can ue thee ampled document to etimate the expected preciion within tratum. Since the incomplete relevance judgment from each tratum i obtained by uniform random ampling, thi expected preciion can be computed a r k /n k. Note that in computing the expected preciion in tratum, we may face the problem of not having ampled any document from thi tratum that are above the current relevant document at rank k. Adapting the ame idea ued in, we employ Lindtone moothing [6] to avoid thi problem. Therefore, expected preciion above rank k can be computed a: E[ d P C above k ] = X N k k r k n k + ɛ + 2ɛ It i eay to ee that when complete judgment are available, x i exactly equal to average preciion (ignoring the moothing effect). Further, note that i a particular intantiation of thi formula with a ingle tratum ued. Overall, the advantage and real power of the decribed tratified random ampling and the derived AP etimator, x, i the fact that it combine the effectivene of the ampling method propoed by Alam et al. [] by employing tratification of the document and thu better utilization of the judgment effort with the implicity of by employing random ampling within each tratum. 3. Inferred AP in TREC Terabyte A mentioned, x can be ued with a large variety of ampling trategy. In thi ection, we focu on the ampling trategy ued in TREC Terabyte 26 [4] and we how that () x i highly effective at etimating average preciion and (2) it better utilize the judgment effort compared to. Firt, let briefly conider the ampling trategy ued in TREC Terabyte 26. In thi track, three different et of relevance judgment were formed, with only two of them being ued for evaluation purpoe. Out of thee two et, the firt et of judgment, contructed by the traditional depth-5 pooling trategy, wa ued to obtain a rough idea of the ytem average preciion. The econd et of judgment wa contructed uing random ampling in uch a way that there are more document judged from topic that are more likely to have retrieved more relevant document. Since, in Terabyte track, the ize of the document collection i very large, the ytem may continue retrieving relevant document even at high rank (deeper in the lit). Thi et of judgment wa created to obtain an etimate of average preciion if complete judgment were preent. To etimate average preciion, wa ued a the evaluation meaure. Since, by deign, aume that the et of relevance judgment i a random ubet of complete judgment, even though the entire depth-5 pool wa judged, wa computed only uing the random ample of judgment (econd et) without utilizing judgment from the depth-5 pool. Therefore, many relevance judgment were not ued even though they were available. Note that x can eaily handle thi etup and it could be ued to utilize all the judgment, obtaining better etimate of average preciion. To tet how x compare with we imulate the ampling trategy ued in TREC Terabyte 6 on data from

5 . The TREC Terabyte data wa not ued due to the fact that in TREC Terabyte the actual value of average preciion i not known ince complete judgment are not available. To imulate the etup ued in TREC Terabyte, we firt form different depth-k pool where k {, 2, 3, 4, 5,, 2, 3, 4, 5} and obtain judgment for all document in each one of thee pool. Then, for each value of k, we compute the total number of document that are in the depth-k pool and we randomly ample equal number of document from the complete judgment et 2 excluding the depth-k pool. After forming thee two et of judgment (depth-k and random) we combine them and compute x on thee combined judgment. Thi etup exactly correpond to a ampling trategy where complete judgment are divided into two trata and judgment are formed by uniformly and independently ampling within each tratum. Note that in TREC, there are ome ytem that were ubmitted but that did not contribute to the pool. To further evaluate the quality of our etimator in term of their robutne for evaluating the quality of uneen ytem (ytem that did not contribute to the pool), when we form the incomplete relevance judgment, we only conider the ytem that contribute to the pool but we compute the x etimate for all ubmitted ytem. Figure 2 demontrate how x computed uing judgment generated by combining (left) depth-, (middle) depth- 5 and (right) depth- pool with equal number of randomly ampled judgment compare with the actual AP. Each of thee depth correpond to judging 23.%, 2.7% and 3.5% of the entire pool, repectively. The plot report the RMS error (how accurate are the etimated value?), the Kendall τ value (how accurate are the etimated ranking of ytem?) and the linear correlation coefficient, ρ, (how well do the etimated value fit in a traight line compared to the actual value?). The dot ign in the figure refer to the ytem that were ued to create the original pool and the plu ign refer to the ytem that did not contribute to the pool. The reult illutrated in thee plot reinforce our claim that x i an unbiaed etimator of average preciion. Furthermore, it can be een that the meaure can reliably be ued to evaluate the quality of ytem that were not ued to create the initial ample, hence the meaure i robut to evaluating the quality of uneen ytem. Figure 3 illutrate how x computed on a non-random judgment et compare with computed on a random judgment et for variou level of incompletene. In a imilar manner to the experimental etup of the original work, for each value of k, we generated ten different ample trial according to the procedure decribed in the previou paragraph, and for each one of the ten trial we computed the x for all ytem. Then, all three tatitic were computed for each one of the trial and the average of thee tatitic over all ten trial were reported for different level of judgment incompletene. Uing the ame procedure, we alo created ten different ample trial where the ample were generated by merely randomly ampling the judgment 2 Throughout thi paper, we aume that the complete judgment et correpond to the depth- pool a the judgment we have are formed uing depth- pool and auming the remaining document are nonrelevant. et and the value were computed on them. For comparion purpoe, to how how the original verion of behave when thi randomne aumption i violated, we alo include run on the ame judgment et a extended (marked a depth+random judgment in the Figure). It can be een that for all level of incompletene, in term of all three tatitic, x i much more accurate in etimating average preciion than the other two meaure. We further compared x to the ampling method propoed by Alam et al. []. The robutne of x to incomplete relevance judgment i comparable to (and in ome cae even better than) thi method. (Thee reult were omitted due to pace limitation.) 4. ESTIMATION OF NDCG WITH INCOMPLETE JUDGMENTS There are different verion of the ndcg metric depending on the dicount factor and the gain aociated with relevance grade, etc. In thi paper, we adopt the verion of ndcg in trec_eval. Let I denote a relevance grade and gain(i) the gain aociated with I. Alo, let g, g 2,... g Z be the gain value aociated with the Z document retrieved by a ytem in repone to a query q, uch a g i = gain(i) if the relevance grade of the document in rank i i I. Then, the ndcg value for thi ytem can be computed a, ndcg = DCG DCG I where DCG = ZX g i/ lg(i + ) and DCG I denote the DCG value for an ideal ranked lit for query q. The etimation of ndcg with incomplete judgment can be divided into two part: () Etimating DCG I and (2) Etimating DCG. Then, the DCG and the DCG I value can be replaced by their etimate to obtain the etimated ndcg value Etimating DCG I The normalization factor, DCG I, for a query q can be defined a the maximum poible DCG value over that query. Hence, the etimation of DCG I can be derived in a two-tep proce: () For each relevance grade I uch a gain(i) >, etimate the number of document with that relevance grade; (2) Calculate the DCG value of an optimal lit by auming that in an optimal lit the etimated number of document would be orted (in decending order) by their relevance grade. Uing the ampling trategy decribed in the previou ection, uppoe incomplete relevance judgment were created by diving the complete pool into dijoint et (trata) and randomly picking (ampling) document from each tratum to be judged, poibly with different probability for each tratum. 3 Note that thi aume that E[nDCG] = E[DCG]/E[DCG I], i.e., that DCG I and DCG are independent of each other, which i not necearily the cae. Thi aumption may reult in a mall bia and better etimate of ndcg can be obtained by conidering thi dependence. However, for the ake of implicity, throughout thi paper, we will aume that thee term are independent. i=

6 xinf map v. actual map, 2.7% judgment xinf map v. actual map, 3.5% judgment.5 Pool Sy RMS =.26, τ =.9345, ρ =.9963 Non Pool Sy RMS =.33, τ =.8678, ρ = Pool Sy RMS =.26, τ =.8699, ρ =.982 Non Pool Sy RMS =.53, τ =.848, ρ =.987 extended inferred map extended inferred map actual map actual map Figure 2: TREC-8 mean x when relevance judgment are generated according to depth-, depth-5 and depth- pooling combined with equivalent number of randomly ampled judgment veru mean actual AP..25 InfAP depth + random judgment extended InfAP random judgment Kendall τ Linear correlation coefficient (ρ) RMS error InfAP depth + random judgment.95 InfAP depth + random judgment.5.7 InfAP random judgment InfAP random judgment Figure 3: TREC-8 change in Kendall τ, linear correlation coefficient (ρ), and RMS error of x and a the judgment et are reduced when half of the judgment are generated according to depth pooling and the other half i a random ubet of complete judgment and of inferred AP when the judgment are a random ubet of complete judgment. For each tratum, let r (I) be the number of ampled document with relevance grade I, let n be the total number of document ampled from trata and N be the total number of document that fall in trata. Since the n document are ampled uniformly from trata, the etimated number of document with relevance grade I within thi trata can be computed a ˆR (I) = r(i) N n Then, the expected number of document with relevance grade I within the complete pool can be computed a ˆR(I) = X ˆR (I) Once thee etimate are obtained, one can etimate DCG I. 4.2 Etimating DCG Given Z document retrieved by a earch engine with relevance gain g i for the document at rank i, for each rank g i, define a new variable x i uch a x i = Z i. Then, lg(i+) DCG can be written a the output of the following random experiment:. Pick a document at random from the output of the earch engine, let the rank of thi document be i. 2. Output the value of x i. It i eay to ee that if we have the relevance judgment for all Z document, the expected value of thi random experiment i exactly equal to DCG. Now conider etimating the outcome of thi random experiment when relevance judgment are incomplete. Conider the firt tep of the random experiment, i.e. picking a document at random. Let Z be the number of document in the output of a ytem that fall in tratum. When picking a document at random, with probability Z /Z, we pick a document from tratum. Therefore, the expected value of the above random experiment can be written a: E[DCG] = X Z Z E[xi document at rank i ] Now conider the econd tep of the random experiment, computing the expected value of x i given that the document at rank i fall in trata. Let ampled be the et of ampled document from trata and n be the number of document ampled from thi trata. Since document within tratum are uniformly ampled, the expected value of x i can be computed a E[x i document at rank i ] = X n j ampled x j Once E[DCG I] and E[DCG] are computed, infndcg can then be computed a infndcg = E[DCG]/E[DCG I]. 5. OVERALL RESULTS Until now, we have hown that uing a imilar ampling trategy a the one ued in TREC Terabyte 6 (complete

7 judgment divided into 2 different trata), x i highly accurate. In thi ection, we how that () thi claim i conitent over different TREC for both x and infndcg and that (2) the two meaure can be ued with the complete judgment divided into more than two trata. In order to check (2), we ue a different ampling trategy than the one in Terabyte; we divide the complete judgment et (auming depth- pool i the complete judgment et) into 4 different trata. The firt tratum i the regular depthk pool, fully judged. Intead of randomly ampling equal to the depth-k pool number of judgment from the remainder of the collection, we now divide the ret of the document into three other trata and ditribute the remaining judgment with a ratio of 3:.5: (judge 55% of the document in the top depth tratum, 27% of the document in the middle depth tratum and 8% in the lowet depth tratum). Thi way, more weight i given to judging document retrieved toward the top of the ranked lit of the earch engine. Note, however, that a the number of trata increae, there value of the etimate may lightly deviate from the actual value ince the effect of moothing alo increae (moothing i needed for each tratum). Figure 4 how the quality of x and infndcg (referred a to avoid confuion) computed on thee ample according to Kendall τ and RMS Error tatitic, for, 9 and. For comparion purpoe, the plot alo contain and ndcg (the tandard formula computed on random judgment, auming unjudged document are nonrelevant). Looking at all plot, it can be een that according to both tatitic, uing the ame number of judgment, the extended (x) and infndcg conitently outperform and ndcg on random judgment, repectively. The high RMS error of ndcg on random judgment i due to the fact that ndcg i computed on thee judgment a it i, without aiming at etimating the value of the meaure. 6. CONCLUSIONS In thi work, we extended inferred AP in two different way. Firt, we derived confidence interval for to capture the variability in value. Employing confidence interval enable comparion and eventually ranking of ytem according to their quality meaured by AP with high confidence. Second, we utilized a tratified random ampling trategy to elect document to be judged and extended to handle the non-random ample of judgment. We applied the ame methodology for etimating ndcg in the preence of incomplete non-random judgment. Stratified random ampling combine the effectivene of tratification and thu better utilization of the relevance judgment with the implicity of random ampling. We howed that x and infndcg are more accurate than and ndcg on equal number of random ample. Note that the ampling trategy (i.e. the number of trata, the ize of each tratum and the ampling percentage from each tratum) ued here i rather arbitrary. The confidence interval a decribed in the firt part of thi paper could be ued a an objective function to determine an optimal ampling trategy. The ampling trategy i highly important for the quality of the etimate and identifying an optimal trategy i a point of future reearch. Furthermore, confidence interval a a function of the ample ize could be ued to determine the appropriate number of document to be judged for an accurate MAP etimation which i a point we plan to invetigate. 7. REFERENCES [] J. A. Alam, V. Pavlu, and E. Yilmaz. A tatitical method for ytem evaluation uing incomplete judgment. In Proceeding of the 29th annual international ACM SIGIR conference on Reearch and development in information retrieval, page ACM Pre, Augut 26. [2] T. Bompada, C.-C. Chang, J. Chen, R. Kumar, and R. Shenoy. On the robutne of relevance meaure with incomplete judgment. In SIGIR 7: Proceeding of the 3th annual international ACM SIGIR conference on Reearch and development in information retrieval, page , New York, NY, USA, 27. ACM Pre. [3] C. Buckley and E. M. Voorhee. Retrieval evaluation with incomplete information. In Proceeding of the 27th Annual International ACM SIGIR Conference on Reearch and Development in Information Retrieval, page 25 32, 24. [4] S. Buttcher, C. Clarke, and I. Soboroff. The TREC 26 terabyte track. In Proceeding of the Fifteenth Text REtrieval Conference (TREC 26), 26. [5] B. Carterette, J. Allan, and R. Sitaraman. Minimal tet collection for retrieval evaluation. In Proceeding of the 29th Annual International ACM SIGIR Conference on Reearch and Development in Information Retrieval, page , 26. [6] S. F. Chen and J. Goodman. An empirical tudy of moothing technique for language modeling. In Proceeding of the Thirty-Fourth Annual Meeting of the Aociation for Computational Linguitic, page 3 38, San Francico, 996. Morgan Kaufmann Publiher. [7] C. Cleverdon. The cranfield tet on index language device. Reading in information retrieval, page 47 59, 997. [8] D. Harman. Overview of the third text REtreival conference (TREC-3). In D. Harman, editor, Overview of the Third Text REtrieval Conference (TREC-3), page 9. U.S. Government Printing Office, Apr [9] K. Järvelin and J. Kekäläinen. Cumulated gain-baed evaluation of ir technique. ACM Tranaction on Information Sytem, 2(4): , 22. [] W. Kraaij, P. Over, and A. Smeaton. TRECVID 26 - an introduction. In TREC Video Retrieval Evaluation Online Proceeding, 26. [] A. Moffat, W. Webber, and J. Zobel. Strategic ytem comparion via targeted relevance judgment. In SIGIR 7: Proceeding of the 3th annual international ACM SIGIR conference on Reearch and development in information retrieval, page , New York, NY, USA, 27. ACM. [2] T. Sakai. Alternative to bpref. In SIGIR 7: Proceeding of the 3th annual international ACM SIGIR conference on Reearch and development in information retrieval, page 7 78, New York, NY, USA, 27. ACM Pre. [3] I. Soboroff. A comparion of pooled and ampled relevance judgment. In SIGIR 7: Proceeding of the 3th annual international ACM SIGIR conference on Reearch and development in information retrieval, page , New York, NY, USA, 27. ACM Pre. [4] E. Yilmaz and J. A. Alam. Etimating average preciion with incomplete and imperfect judgment. In Proceeding of the Fifteenth ACM International Conference on Information and Knowledge Management. ACM Pre, November 26. APPENDIX Let d be a ample of cut-off level at relevant document. According to the Law of Total Variance, the variance in can be calculated a, var[] = var[e[ d ]] + E[var[ d ]] Let conider the firt term of the right-hand ide of the above equation, which correpond to the variance due to ampling cut-off level.

8 5432 TREC 9 TREC Kendall τ.85 Kendall τ.9.85 Kendall τ percentage of pool judged percentage of pool judged percentage of pool judged TREC TREC.3.25 RMS error RMS error.5. RMS error Figure 4: Comparion of extended inferred map, (extended) mean inferred ndcg, inferred map and mean ndcg on random judgment, uing Kendall τ (firt row) and RMS error (lat row) for, 9 and. Let r the number of relevant document in d. Then, the conditional expectation of i, E[ d ] = X E[ P r d C k d ] = X P C k r k d k d where P d C k and P C k denote the etimated and actual preciion at cut-off k, repectively. Thu, 2 3 var[e[ d ]] = var 4 X P C k 5 = ( p) σ2 r r k d where p% i the ampling percentage of document from the entire depth- pool and σ 2 i the actual variance among the preciion value at all cut-off of relevant document and P it can be etimated by, (d P C k d k ) 2 /(r ). Now, let conider the econd term of the right-hand ide of the equation deduced by the Law of Total Variance, that i the variance due to ampling document above a cut-off level in order to etimate the preciion at that cut-off level, var[ d ] = var 4 X P d Ck 5 = r r var 4 X dp C 2 k 5 k d k d Conidering d P C k independent from each other If we conider preciion at different cut-off level independent from each other the variance of for a given et of ampled cut-off level depend on the ummation of the preciion variance at each individual cut-off level, var[ d ] = X var[ P r d C 2 k d ] k d The preciion at cut-off i alway and therefore the variance i. Moreover, the preciion at relevant document not in the retrieved lit i alway aumed to be and therefore, the variance at thoe cut-off level i alo. In all other cae dp C k i calculated a, P d C k = /k + ((k )/k) dp C above k and therefore, «2 var[ P d k C k d ] = var[ P k d C above k ] Let r k and n k be the number of relevant document and total number of document ampled above cut-off k, repectively and let d k be the number of document in the depth- pool above cut-off k. The preciion above cut-off k i etimated by 4, P d C k = d k rk k n k, which follow a hypergeometric ditribution and it variance can be calculated a, var[ P d p( p) C k d ] = n k «n «k d k By conidering the expected value of var[ d ] over all ample of cut-off level we get, P k E[var[ d ]] = d var[ P d C k d ] r 2 Conidering d P C k dependent to each other If we do not conider preciion at different cut-off level independent from each other the covariance between preciion can be calculated a, cov[ d P C k, d P C m] = k m var[d P C k ] where k < m 4 For implicity reaon we ignore the effect of moothing that i introduced in the formula of. Smoothing wa conidered in all experiment ran and it wa oberved that the effect of moothing in variance i negligible.

A technical guide to 2014 key stage 2 to key stage 4 value added measures

A technical guide to 2014 key stage 2 to key stage 4 value added measures A technical guide to 2014 key tage 2 to key tage 4 value added meaure CONTENTS Introduction: PAGE NO. What i value added? 2 Change to value added methodology in 2014 4 Interpretation: Interpreting chool

More information

Assessing the Discriminatory Power of Credit Scores

Assessing the Discriminatory Power of Credit Scores Aeing the Dicriminatory Power of Credit Score Holger Kraft 1, Gerald Kroiandt 1, Marlene Müller 1,2 1 Fraunhofer Intitut für Techno- und Wirtchaftmathematik (ITWM) Gottlieb-Daimler-Str. 49, 67663 Kaierlautern,

More information

DISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS

DISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS DISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS Chritopher V. Kopek Department of Computer Science Wake Foret Univerity Winton-Salem, NC, 2709 Email: kopekcv@gmail.com

More information

DISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS. G. Chapman J. Cleese E. Idle

DISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS. G. Chapman J. Cleese E. Idle DISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS G. Chapman J. Cleee E. Idle ABSTRACT Content matching i a neceary component of any ignature-baed network Intruion Detection

More information

Optical Illusion. Sara Bolouki, Roger Grosse, Honglak Lee, Andrew Ng

Optical Illusion. Sara Bolouki, Roger Grosse, Honglak Lee, Andrew Ng Optical Illuion Sara Bolouki, Roger Groe, Honglak Lee, Andrew Ng. Introduction The goal of thi proect i to explain ome of the illuory phenomena uing pare coding and whitening model. Intead of the pare

More information

T-test for dependent Samples. Difference Scores. The t Test for Dependent Samples. The t Test for Dependent Samples. s D

T-test for dependent Samples. Difference Scores. The t Test for Dependent Samples. The t Test for Dependent Samples. s D The t Tet for ependent Sample T-tet for dependent Sample (ak.a., Paired ample t-tet, Correlated Group eign, Within- Subject eign, Repeated Meaure,.. Repeated-Meaure eign When you have two et of core from

More information

Queueing systems with scheduled arrivals, i.e., appointment systems, are typical for frontal service systems,

Queueing systems with scheduled arrivals, i.e., appointment systems, are typical for frontal service systems, MANAGEMENT SCIENCE Vol. 54, No. 3, March 28, pp. 565 572 in 25-199 ein 1526-551 8 543 565 inform doi 1.1287/mnc.17.82 28 INFORMS Scheduling Arrival to Queue: A Single-Server Model with No-Show INFORMS

More information

1 Introduction. Reza Shokri* Privacy Games: Optimal User-Centric Data Obfuscation

1 Introduction. Reza Shokri* Privacy Games: Optimal User-Centric Data Obfuscation Proceeding on Privacy Enhancing Technologie 2015; 2015 (2):1 17 Reza Shokri* Privacy Game: Optimal Uer-Centric Data Obfucation Abtract: Conider uer who hare their data (e.g., location) with an untruted

More information

Multi-Objective Optimization for Sponsored Search

Multi-Objective Optimization for Sponsored Search Multi-Objective Optimization for Sponored Search Yilei Wang 1,*, Bingzheng Wei 2, Jun Yan 2, Zheng Chen 2, Qiao Du 2,3 1 Yuanpei College Peking Univerity Beijing, China, 100871 (+86)15120078719 wangyileipku@gmail.com

More information

A note on profit maximization and monotonicity for inbound call centers

A note on profit maximization and monotonicity for inbound call centers A note on profit maximization and monotonicity for inbound call center Ger Koole & Aue Pot Department of Mathematic, Vrije Univeriteit Amterdam, The Netherland 23rd December 2005 Abtract We conider an

More information

A Note on Profit Maximization and Monotonicity for Inbound Call Centers

A Note on Profit Maximization and Monotonicity for Inbound Call Centers OPERATIONS RESEARCH Vol. 59, No. 5, September October 2011, pp. 1304 1308 in 0030-364X ein 1526-5463 11 5905 1304 http://dx.doi.org/10.1287/opre.1110.0990 2011 INFORMS TECHNICAL NOTE INFORMS hold copyright

More information

A Spam Message Filtering Method: focus on run time

A Spam Message Filtering Method: focus on run time , pp.29-33 http://dx.doi.org/10.14257/atl.2014.76.08 A Spam Meage Filtering Method: focu on run time Sin-Eon Kim 1, Jung-Tae Jo 2, Sang-Hyun Choi 3 1 Department of Information Security Management 2 Department

More information

THE IMPACT OF MULTIFACTORIAL GENETIC DISORDERS ON CRITICAL ILLNESS INSURANCE: A SIMULATION STUDY BASED ON UK BIOBANK ABSTRACT KEYWORDS

THE IMPACT OF MULTIFACTORIAL GENETIC DISORDERS ON CRITICAL ILLNESS INSURANCE: A SIMULATION STUDY BASED ON UK BIOBANK ABSTRACT KEYWORDS THE IMPACT OF MULTIFACTORIAL GENETIC DISORDERS ON CRITICAL ILLNESS INSURANCE: A SIMULATION STUDY BASED ON UK BIOBANK BY ANGUS MACDONALD, DELME PRITCHARD AND PRADIP TAPADAR ABSTRACT The UK Biobank project

More information

Redesigning Ratings: Assessing the Discriminatory Power of Credit Scores under Censoring

Redesigning Ratings: Assessing the Discriminatory Power of Credit Scores under Censoring Redeigning Rating: Aeing the Dicriminatory Power of Credit Score under Cenoring Holger Kraft, Gerald Kroiandt, Marlene Müller Fraunhofer Intitut für Techno- und Wirtchaftmathematik (ITWM) Thi verion: June

More information

Partial optimal labeling search for a NP-hard subclass of (max,+) problems

Partial optimal labeling search for a NP-hard subclass of (max,+) problems Partial optimal labeling earch for a NP-hard ubcla of (max,+) problem Ivan Kovtun International Reearch and Training Center of Information Technologie and Sytem, Kiev, Uraine, ovtun@image.iev.ua Dreden

More information

INFORMATION Technology (IT) infrastructure management

INFORMATION Technology (IT) infrastructure management IEEE TRANSACTIONS ON CLOUD COMPUTING, VOL. 2, NO. 1, MAY 214 1 Buine-Driven Long-term Capacity Planning for SaaS Application David Candeia, Ricardo Araújo Santo and Raquel Lope Abtract Capacity Planning

More information

Morningstar Fixed Income Style Box TM Methodology

Morningstar Fixed Income Style Box TM Methodology Morningtar Fixed Income Style Box TM Methodology Morningtar Methodology Paper Augut 3, 00 00 Morningtar, Inc. All right reerved. The information in thi document i the property of Morningtar, Inc. Reproduction

More information

Unit 11 Using Linear Regression to Describe Relationships

Unit 11 Using Linear Regression to Describe Relationships Unit 11 Uing Linear Regreion to Decribe Relationhip Objective: To obtain and interpret the lope and intercept of the leat quare line for predicting a quantitative repone variable from a quantitative explanatory

More information

Mixed Method of Model Reduction for Uncertain Systems

Mixed Method of Model Reduction for Uncertain Systems SERBIAN JOURNAL OF ELECTRICAL ENGINEERING Vol 4 No June Mixed Method of Model Reduction for Uncertain Sytem N Selvaganean Abtract: A mixed method for reducing a higher order uncertain ytem to a table reduced

More information

A Resolution Approach to a Hierarchical Multiobjective Routing Model for MPLS Networks

A Resolution Approach to a Hierarchical Multiobjective Routing Model for MPLS Networks A Reolution Approach to a Hierarchical Multiobjective Routing Model for MPLS Networ Joé Craveirinha a,c, Rita Girão-Silva a,c, João Clímaco b,c, Lúcia Martin a,c a b c DEEC-FCTUC FEUC INESC-Coimbra International

More information

TRADING rules are widely used in financial market as

TRADING rules are widely used in financial market as Complex Stock Trading Strategy Baed on Particle Swarm Optimization Fei Wang, Philip L.H. Yu and David W. Cheung Abtract Trading rule have been utilized in the tock market to make profit for more than a

More information

Measuring the Ability of Score Distributions to Model Relevance

Measuring the Ability of Score Distributions to Model Relevance Meauring the Ability of Score Ditribution to Model Relevance Ronan Cummin Department of Information Technology National Univerity of Ireland, Galway ronan.cummin@nuigalway.ie Abtract. Modelling the core

More information

Scheduling of Jobs and Maintenance Activities on Parallel Machines

Scheduling of Jobs and Maintenance Activities on Parallel Machines Scheduling of Job and Maintenance Activitie on Parallel Machine Chung-Yee Lee* Department of Indutrial Engineering Texa A&M Univerity College Station, TX 77843-3131 cylee@ac.tamu.edu Zhi-Long Chen** Department

More information

Tap Into Smartphone Demand: Mobile-izing Enterprise Websites by Using Flexible, Open Source Platforms

Tap Into Smartphone Demand: Mobile-izing Enterprise Websites by Using Flexible, Open Source Platforms Tap Into Smartphone Demand: Mobile-izing Enterprie Webite by Uing Flexible, Open Source Platform acquia.com 888.922.7842 1.781.238.8600 25 Corporate Drive, Burlington, MA 01803 Tap Into Smartphone Demand:

More information

1) Assume that the sample is an SRS. The problem state that the subjects were randomly selected.

1) Assume that the sample is an SRS. The problem state that the subjects were randomly selected. 12.1 Homework for t Hypothei Tet 1) Below are the etimate of the daily intake of calcium in milligram for 38 randomly elected women between the age of 18 and 24 year who agreed to participate in a tudy

More information

REDUCTION OF TOTAL SUPPLY CHAIN CYCLE TIME IN INTERNAL BUSINESS PROCESS OF REAMER USING DOE AND TAGUCHI METHODOLOGY. Abstract. 1.

REDUCTION OF TOTAL SUPPLY CHAIN CYCLE TIME IN INTERNAL BUSINESS PROCESS OF REAMER USING DOE AND TAGUCHI METHODOLOGY. Abstract. 1. International Journal of Advanced Technology & Engineering Reearch (IJATER) REDUCTION OF TOTAL SUPPLY CHAIN CYCLE TIME IN INTERNAL BUSINESS PROCESS OF REAMER USING DOE AND Abtract TAGUCHI METHODOLOGY Mr.

More information

TIME SERIES ANALYSIS AND TRENDS BY USING SPSS PROGRAMME

TIME SERIES ANALYSIS AND TRENDS BY USING SPSS PROGRAMME TIME SERIES ANALYSIS AND TRENDS BY USING SPSS PROGRAMME RADMILA KOCURKOVÁ Sileian Univerity in Opava School of Buine Adminitration in Karviná Department of Mathematical Method in Economic Czech Republic

More information

Name: SID: Instructions

Name: SID: Instructions CS168 Fall 2014 Homework 1 Aigned: Wedneday, 10 September 2014 Due: Monday, 22 September 2014 Name: SID: Dicuion Section (Day/Time): Intruction - Submit thi homework uing Pandagrader/GradeScope(http://www.gradecope.com/

More information

Performance of Multiple TFRC in Heterogeneous Wireless Networks

Performance of Multiple TFRC in Heterogeneous Wireless Networks Performance of Multiple TFRC in Heterogeneou Wirele Network 1 Hyeon-Jin Jeong, 2 Seong-Sik Choi 1, Firt Author Computer Engineering Department, Incheon National Univerity, oaihjj@incheon.ac.kr *2,Correponding

More information

Review of Multiple Regression Richard Williams, University of Notre Dame, http://www3.nd.edu/~rwilliam/ Last revised January 13, 2015

Review of Multiple Regression Richard Williams, University of Notre Dame, http://www3.nd.edu/~rwilliam/ Last revised January 13, 2015 Review of Multiple Regreion Richard William, Univerity of Notre Dame, http://www3.nd.edu/~rwilliam/ Lat revied January 13, 015 Aumption about prior nowledge. Thi handout attempt to ummarize and yntheize

More information

Progress 8 measure in 2016, 2017, and 2018. Guide for maintained secondary schools, academies and free schools

Progress 8 measure in 2016, 2017, and 2018. Guide for maintained secondary schools, academies and free schools Progre 8 meaure in 2016, 2017, and 2018 Guide for maintained econdary chool, academie and free chool July 2016 Content Table of figure 4 Summary 5 A ummary of Attainment 8 and Progre 8 5 Expiry or review

More information

Proceedings of Power Tech 2007, July 1-5, Lausanne

Proceedings of Power Tech 2007, July 1-5, Lausanne Second Order Stochatic Dominance Portfolio Optimization for an Electric Energy Company M.-P. Cheong, Student Member, IEEE, G. B. Sheble, Fellow, IEEE, D. Berleant, Senior Member, IEEE and C.-C. Teoh, Student

More information

Turbulent Mixing and Chemical Reaction in Stirred Tanks

Turbulent Mixing and Chemical Reaction in Stirred Tanks Turbulent Mixing and Chemical Reaction in Stirred Tank André Bakker Julian B. Faano Blend time and chemical product ditribution in turbulent agitated veel can be predicted with the aid of Computational

More information

Cluster-Aware Cache for Network Attached Storage *

Cluster-Aware Cache for Network Attached Storage * Cluter-Aware Cache for Network Attached Storage * Bin Cai, Changheng Xie, and Qiang Cao National Storage Sytem Laboratory, Department of Computer Science, Huazhong Univerity of Science and Technology,

More information

6. Friction, Experiment and Theory

6. Friction, Experiment and Theory 6. Friction, Experiment and Theory The lab thi wee invetigate the rictional orce and the phyical interpretation o the coeicient o riction. We will mae ue o the concept o the orce o gravity, the normal

More information

Bi-Objective Optimization for the Clinical Trial Supply Chain Management

Bi-Objective Optimization for the Clinical Trial Supply Chain Management Ian David Lockhart Bogle and Michael Fairweather (Editor), Proceeding of the 22nd European Sympoium on Computer Aided Proce Engineering, 17-20 June 2012, London. 2012 Elevier B.V. All right reerved. Bi-Objective

More information

Acceleration-Displacement Crash Pulse Optimisation A New Methodology to Optimise Vehicle Response for Multiple Impact Speeds

Acceleration-Displacement Crash Pulse Optimisation A New Methodology to Optimise Vehicle Response for Multiple Impact Speeds Acceleration-Diplacement Crah Pule Optimiation A New Methodology to Optimie Vehicle Repone for Multiple Impact Speed D. Gildfind 1 and D. Ree 2 1 RMIT Univerity, Department of Aeropace Engineering 2 Holden

More information

CHARACTERISTICS OF WAITING LINE MODELS THE INDICATORS OF THE CUSTOMER FLOW MANAGEMENT SYSTEMS EFFICIENCY

CHARACTERISTICS OF WAITING LINE MODELS THE INDICATORS OF THE CUSTOMER FLOW MANAGEMENT SYSTEMS EFFICIENCY Annale Univeritati Apuleni Serie Oeconomica, 2(2), 200 CHARACTERISTICS OF WAITING LINE MODELS THE INDICATORS OF THE CUSTOMER FLOW MANAGEMENT SYSTEMS EFFICIENCY Sidonia Otilia Cernea Mihaela Jaradat 2 Mohammad

More information

Independent Samples T- test

Independent Samples T- test Independent Sample T- tet With previou tet, we were intereted in comparing a ingle ample with a population With mot reearch, you do not have knowledge about the population -- you don t know the population

More information

Profitability of Loyalty Programs in the Presence of Uncertainty in Customers Valuations

Profitability of Loyalty Programs in the Presence of Uncertainty in Customers Valuations Proceeding of the 0 Indutrial Engineering Reearch Conference T. Doolen and E. Van Aken, ed. Profitability of Loyalty Program in the Preence of Uncertainty in Cutomer Valuation Amir Gandomi and Saeed Zolfaghari

More information

Risk Management for a Global Supply Chain Planning under Uncertainty: Models and Algorithms

Risk Management for a Global Supply Chain Planning under Uncertainty: Models and Algorithms Rik Management for a Global Supply Chain Planning under Uncertainty: Model and Algorithm Fengqi You 1, John M. Waick 2, Ignacio E. Gromann 1* 1 Dept. of Chemical Engineering, Carnegie Mellon Univerity,

More information

Maximizing Acceptance Probability for Active Friending in Online Social Networks

Maximizing Acceptance Probability for Active Friending in Online Social Networks Maximizing for Active Friending in Online Social Network De-Nian Yang, Hui-Ju Hung, Wang-Chien Lee, Wei Chen Academia Sinica, Taipei, Taiwan The Pennylvania State Univerity, State College, Pennylvania,

More information

Support Vector Machine Based Electricity Price Forecasting For Electricity Markets utilising Projected Assessment of System Adequacy Data.

Support Vector Machine Based Electricity Price Forecasting For Electricity Markets utilising Projected Assessment of System Adequacy Data. The Sixth International Power Engineering Conference (IPEC23, 27-29 November 23, Singapore Support Vector Machine Baed Electricity Price Forecating For Electricity Maret utiliing Projected Aement of Sytem

More information

Report 4668-1b 30.10.2010. Measurement report. Sylomer - field test

Report 4668-1b 30.10.2010. Measurement report. Sylomer - field test Report 4668-1b Meaurement report Sylomer - field tet Report 4668-1b 2(16) Contet 1 Introduction... 3 1.1 Cutomer... 3 1.2 The ite and purpoe of the meaurement... 3 2 Meaurement... 6 2.1 Attenuation of

More information

CASE STUDY BRIDGE. www.future-processing.com

CASE STUDY BRIDGE. www.future-processing.com CASE STUDY BRIDGE TABLE OF CONTENTS #1 ABOUT THE CLIENT 3 #2 ABOUT THE PROJECT 4 #3 OUR ROLE 5 #4 RESULT OF OUR COLLABORATION 6-7 #5 THE BUSINESS PROBLEM THAT WE SOLVED 8 #6 CHALLENGES 9 #7 VISUAL IDENTIFICATION

More information

Bidding for Representative Allocations for Display Advertising

Bidding for Representative Allocations for Display Advertising Bidding for Repreentative Allocation for Diplay Advertiing Arpita Ghoh, Preton McAfee, Kihore Papineni, and Sergei Vailvitkii Yahoo! Reearch. {arpita, mcafee, kpapi, ergei}@yahoo-inc.com Abtract. Diplay

More information

Queueing Models for Multiclass Call Centers with Real-Time Anticipated Delays

Queueing Models for Multiclass Call Centers with Real-Time Anticipated Delays Queueing Model for Multicla Call Center with Real-Time Anticipated Delay Oualid Jouini Yve Dallery Zeynep Akşin Ecole Centrale Pari Koç Univerity Laboratoire Génie Indutriel College of Adminitrative Science

More information

Computing Location from Ambient FM Radio Signals

Computing Location from Ambient FM Radio Signals Computing Location from Ambient FM Radio Signal Adel Youef Department of Computer Science Univerity of Maryland A.V. William Building College Park, MD 20742 adel@c.umd.edu John Krumm, Ed Miller, Gerry

More information

NETWORK TRAFFIC ENGINEERING WITH VARIED LEVELS OF PROTECTION IN THE NEXT GENERATION INTERNET

NETWORK TRAFFIC ENGINEERING WITH VARIED LEVELS OF PROTECTION IN THE NEXT GENERATION INTERNET Chapter 1 NETWORK TRAFFIC ENGINEERING WITH VARIED LEVELS OF PROTECTION IN THE NEXT GENERATION INTERNET S. Srivatava Univerity of Miouri Kana City, USA hekhar@conrel.ice.umkc.edu S. R. Thirumalaetty now

More information

Towards Control-Relevant Forecasting in Supply Chain Management

Towards Control-Relevant Forecasting in Supply Chain Management 25 American Control Conference June 8-1, 25. Portland, OR, USA WeA7.1 Toward Control-Relevant Forecating in Supply Chain Management Jay D. Schwartz, Daniel E. Rivera 1, and Karl G. Kempf Control Sytem

More information

CASE STUDY ALLOCATE SOFTWARE

CASE STUDY ALLOCATE SOFTWARE CASE STUDY ALLOCATE SOFTWARE allocate caetud y TABLE OF CONTENTS #1 ABOUT THE CLIENT #2 OUR ROLE #3 EFFECTS OF OUR COOPERATION #4 BUSINESS PROBLEM THAT WE SOLVED #5 CHALLENGES #6 WORKING IN SCRUM #7 WHAT

More information

MSc Financial Economics: International Finance. Bubbles in the Foreign Exchange Market. Anne Sibert. Revised Spring 2013. Contents

MSc Financial Economics: International Finance. Bubbles in the Foreign Exchange Market. Anne Sibert. Revised Spring 2013. Contents MSc Financial Economic: International Finance Bubble in the Foreign Exchange Market Anne Sibert Revied Spring 203 Content Introduction................................................. 2 The Mone Market.............................................

More information

RO-BURST: A Robust Virtualization Cost Model for Workload Consolidation over Clouds

RO-BURST: A Robust Virtualization Cost Model for Workload Consolidation over Clouds !111! 111!ttthhh IIIEEEEEEEEE///AAACCCMMM IIInnnttteeerrrnnnaaatttiiiooonnnaaalll SSSyyymmmpppoooiiiuuummm ooonnn CCCllluuuttteeerrr,,, CCClllooouuuddd aaannnddd GGGrrriiiddd CCCooommmpppuuutttiiinnnggg

More information

Ohm s Law. Ohmic relationship V=IR. Electric Power. Non Ohmic devises. Schematic representation. Electric Power

Ohm s Law. Ohmic relationship V=IR. Electric Power. Non Ohmic devises. Schematic representation. Electric Power Ohm Law Ohmic relationhip V=IR Ohm law tate that current through the conductor i directly proportional to the voltage acro it if temperature and other phyical condition do not change. In many material,

More information

A New Optimum Jitter Protection for Conversational VoIP

A New Optimum Jitter Protection for Conversational VoIP Proc. Int. Conf. Wirele Commun., Signal Proceing (Nanjing, China), 5 pp., Nov. 2009 A New Optimum Jitter Protection for Converational VoIP Qipeng Gong, Peter Kabal Electrical & Computer Engineering, McGill

More information

Sector Concentration in Loan Portfolios and Economic Capital. Abstract

Sector Concentration in Loan Portfolios and Economic Capital. Abstract Sector Concentration in Loan Portfolio and Economic Capital Klau Düllmann and Nancy Machelein 2 Thi verion: September 2006 Abtract The purpoe of thi paper i to meaure the potential impact of buine-ector

More information

Availability of WDM Multi Ring Networks

Availability of WDM Multi Ring Networks Paper Availability of WDM Multi Ring Network Ivan Rado and Katarina Rado H d.o.o. Motar, Motar, Bonia and Herzegovina Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, Univerity

More information

A Review On Software Testing In SDlC And Testing Tools

A Review On Software Testing In SDlC And Testing Tools www.ijec.in International Journal Of Engineering And Computer Science ISSN:2319-7242 Volume - 3 Iue -9 September, 2014 Page No. 8188-8197 A Review On Software Teting In SDlC And Teting Tool T.Amruthavalli*,

More information

Combining Statistics and Semantics via Ensemble Model for Document Clustering

Combining Statistics and Semantics via Ensemble Model for Document Clustering ombining tatitic and emantic via Enemble Model or Document lutering amah Jamal Fodeh Michigan tate Univerity Eat Laning, MI, 48824 odeham@mu.edu William F Punch Michigan tate Univerity Eat Laning, MI,

More information

G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences

G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences Behavior Reearch Method 007, 39 (), 75-9 G*Power 3: A flexible tatitical power analyi program for the ocial, behavioral, and biomedical cience FRAZ FAUL Chritian-Albrecht-Univerität Kiel, Kiel, Germany

More information

Module 8. Three-phase Induction Motor. Version 2 EE IIT, Kharagpur

Module 8. Three-phase Induction Motor. Version 2 EE IIT, Kharagpur Module 8 Three-phae Induction Motor Verion EE IIT, Kharagpur Leon 33 Different Type of Starter for Induction Motor (IM Verion EE IIT, Kharagpur Inructional Objective Need of uing arter for Induction motor

More information

Evaluating Teaching in Higher Education. September 2008. Bruce A. Weinberg The Ohio State University *, IZA, and NBER weinberg.27@osu.

Evaluating Teaching in Higher Education. September 2008. Bruce A. Weinberg The Ohio State University *, IZA, and NBER weinberg.27@osu. Evaluating Teaching in Higher Education September 2008 Bruce A. Weinberg The Ohio State Univerity *, IZA, and NBER weinberg.27@ou.edu Belton M. Fleiher The Ohio State Univerity * and IZA fleiher.1@ou.edu

More information

Utility-Based Flow Control for Sequential Imagery over Wireless Networks

Utility-Based Flow Control for Sequential Imagery over Wireless Networks Utility-Baed Flow Control for Sequential Imagery over Wirele Networ Tomer Kihoni, Sara Callaway, and Mar Byer Abtract Wirele enor networ provide a unique et of characteritic that mae them uitable for building

More information

QUANTIFYING THE BULLWHIP EFFECT IN THE SUPPLY CHAIN OF SMALL-SIZED COMPANIES

QUANTIFYING THE BULLWHIP EFFECT IN THE SUPPLY CHAIN OF SMALL-SIZED COMPANIES Sixth LACCEI International Latin American and Caribbean Conference for Engineering and Technology (LACCEI 2008) Partnering to Succe: Engineering, Education, Reearch and Development June 4 June 6 2008,

More information

INTERACTIVE TOOL FOR ANALYSIS OF TIME-DELAY SYSTEMS WITH DEAD-TIME COMPENSATORS

INTERACTIVE TOOL FOR ANALYSIS OF TIME-DELAY SYSTEMS WITH DEAD-TIME COMPENSATORS INTERACTIVE TOOL FOR ANALYSIS OF TIMEDELAY SYSTEMS WITH DEADTIME COMPENSATORS Joé Lui Guzmán, Pedro García, Tore Hägglund, Sebatián Dormido, Pedro Alberto, Manuel Berenguel Dep. de Lenguaje y Computación,

More information

Laureate Network Products & Services Copyright 2013 Laureate Education, Inc.

Laureate Network Products & Services Copyright 2013 Laureate Education, Inc. Laureate Network Product & Service Copyright 2013 Laureate Education, Inc. KEY Coure Name Laureate Faculty Development...3 Laureate Englih Program...9 Language Laureate Signature Product...12 Length Laureate

More information

Control of Wireless Networks with Flow Level Dynamics under Constant Time Scheduling

Control of Wireless Networks with Flow Level Dynamics under Constant Time Scheduling Control of Wirele Network with Flow Level Dynamic under Contant Time Scheduling Long Le and Ravi R. Mazumdar Department of Electrical and Computer Engineering Univerity of Waterloo,Waterloo, ON, Canada

More information

DUE to the small size and low cost of a sensor node, a

DUE to the small size and low cost of a sensor node, a 1992 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015 A Networ Coding Baed Energy Efficient Data Bacup in Survivability-Heterogeneou Senor Networ Jie Tian, Tan Yan, and Guiling Wang

More information

Exploiting A Support-based Upper Bound of Pearson s Correlation Coefficient for Efficiently Identifying Strongly Correlated Pairs

Exploiting A Support-based Upper Bound of Pearson s Correlation Coefficient for Efficiently Identifying Strongly Correlated Pairs Exploiting A Support-baed Upper Bound of Pearon Correlation Coefficient for Efficiently Identifying Strongly Correlated Pair Hui Xiong Computer Science Univerity of Minneota huix@c.umn.edu Shahi Shekhar

More information

Performance of a Browser-Based JavaScript Bandwidth Test

Performance of a Browser-Based JavaScript Bandwidth Test Performance of a Brower-Baed JavaScript Bandwidth Tet David A. Cohen II May 7, 2013 CP SC 491/H495 Abtract An exiting brower-baed bandwidth tet written in JavaScript wa modified for the purpoe of further

More information

Assigning Tasks for Efficiency in Hadoop

Assigning Tasks for Efficiency in Hadoop Aigning Tak for Efficiency in Hadoop [Extended Abtract] Michael J. Ficher Computer Science Yale Univerity P.O. Box 208285 New Haven, CT, USA michael.ficher@yale.edu Xueyuan Su Computer Science Yale Univerity

More information

Group Mutual Exclusion Based on Priorities

Group Mutual Exclusion Based on Priorities Group Mutual Excluion Baed on Prioritie Karina M. Cenci Laboratorio de Invetigación en Sitema Ditribuido Univeridad Nacional del Sur Bahía Blanca, Argentina kmc@c.un.edu.ar and Jorge R. Ardenghi Laboratorio

More information

AN OVERVIEW ON CLUSTERING METHODS

AN OVERVIEW ON CLUSTERING METHODS IOSR Journal Engineering AN OVERVIEW ON CLUSTERING METHODS T. Soni Madhulatha Aociate Preor, Alluri Intitute Management Science, Warangal. ABSTRACT Clutering i a common technique for tatitical data analyi,

More information

Unobserved Heterogeneity and Risk in Wage Variance: Does Schooling Provide Earnings Insurance?

Unobserved Heterogeneity and Risk in Wage Variance: Does Schooling Provide Earnings Insurance? TI 011-045/3 Tinbergen Intitute Dicuion Paper Unoberved Heterogeneity and Rik in Wage Variance: Doe Schooling Provide Earning Inurance? Jacopo Mazza Han van Ophem Joop Hartog * Univerity of Amterdam; *

More information

How Enterprises Can Build Integrated Digital Marketing Experiences Using Drupal

How Enterprises Can Build Integrated Digital Marketing Experiences Using Drupal How Enterprie Can Build Integrated Digital Marketing Experience Uing Drupal acquia.com 888.922.7842 1.781.238.8600 25 Corporate Drive, Burlington, MA 01803 How Enterprie Can Build Integrated Digital Marketing

More information

POSSIBILITIES OF INDIVIDUAL CLAIM RESERVE RISK MODELING

POSSIBILITIES OF INDIVIDUAL CLAIM RESERVE RISK MODELING POSSIBILITIES OF INDIVIDUAL CLAIM RESERVE RISK MODELING Pavel Zimmermann * 1. Introduction A ignificant increae in demand for inurance and financial rik quantification ha occurred recently due to the fact

More information

MBA 570x Homework 1 Due 9/24/2014 Solution

MBA 570x Homework 1 Due 9/24/2014 Solution MA 570x Homework 1 Due 9/24/2014 olution Individual work: 1. Quetion related to Chapter 11, T Why do you think i a fund of fund market for hedge fund, but not for mutual fund? Anwer: Invetor can inexpenively

More information

Project Management Basics

Project Management Basics Project Management Baic A Guide to undertanding the baic component of effective project management and the key to ucce 1 Content 1.0 Who hould read thi Guide... 3 1.1 Overview... 3 1.2 Project Management

More information

Growing Self-Organizing Maps for Surface Reconstruction from Unstructured Point Clouds

Growing Self-Organizing Maps for Surface Reconstruction from Unstructured Point Clouds Growing Self-Organizing Map for Surface Recontruction from Untructured Point Cloud Renata L. M. E. do Rêgo, Aluizio F. R. Araújo, and Fernando B.de Lima Neto Abtract Thi work introduce a new method for

More information

Return on Investment and Effort Expenditure in the Software Development Environment

Return on Investment and Effort Expenditure in the Software Development Environment International Journal of Applied Information ytem (IJAI) IN : 2249-0868 Return on Invetment and Effort Expenditure in the oftware Development Environment Dineh Kumar aini Faculty of Computing and IT, ohar

More information

Brand Equity Net Promoter Scores Versus Mean Scores. Which Presents a Clearer Picture For Action? A Non-Elite Branded University Example.

Brand Equity Net Promoter Scores Versus Mean Scores. Which Presents a Clearer Picture For Action? A Non-Elite Branded University Example. Brand Equity Net Promoter Score Veru Mean Score. Which Preent a Clearer Picture For Action? A Non-Elite Branded Univerity Example Ann Miti, Swinburne Univerity of Technology Patrick Foley, Victoria Univerity

More information

Improving the Performance of Web Service Recommenders Using Semantic Similarity

Improving the Performance of Web Service Recommenders Using Semantic Similarity Improving the Performance of Web Service Recommender Uing Semantic Similarity Juan Manuel Adán-Coello, Carlo Miguel Tobar, Yang Yuming Faculdade de Engenharia de Computação, Pontifícia Univeridade Católica

More information

Mobility Improves Coverage of Sensor Networks

Mobility Improves Coverage of Sensor Networks Mobility Improve Coverage of Senor Networ Benyuan Liu Dept. of Computer Science Univerity of Maachuett-Lowell Lowell, MA 1854 Peter Bra Dept. of Computer Science City College of New Yor New Yor, NY 131

More information

Mobile Network Configuration for Large-scale Multimedia Delivery on a Single WLAN

Mobile Network Configuration for Large-scale Multimedia Delivery on a Single WLAN Mobile Network Configuration for Large-cale Multimedia Delivery on a Single WLAN Huigwang Je, Dongwoo Kwon, Hyeonwoo Kim, and Hongtaek Ju Dept. of Computer Engineering Keimyung Univerity Daegu, Republic

More information

SCM- integration: organiational, managerial and technological iue M. Caridi 1 and A. Sianei 2 Dipartimento di Economia e Produzione, Politecnico di Milano, Italy E-mail: maria.caridi@polimi.it Itituto

More information

SELF-MANAGING PERFORMANCE IN APPLICATION SERVERS MODELLING AND DATA ARCHITECTURE

SELF-MANAGING PERFORMANCE IN APPLICATION SERVERS MODELLING AND DATA ARCHITECTURE SELF-MANAGING PERFORMANCE IN APPLICATION SERVERS MODELLING AND DATA ARCHITECTURE RAVI KUMAR G 1, C.MUTHUSAMY 2 & A.VINAYA BABU 3 1 HP Bangalore, Reearch Scholar JNTUH, Hyderabad, India, 2 Yahoo, Bangalore,

More information

Risk-Sharing within Families: Evidence from the Health and Retirement Study

Risk-Sharing within Families: Evidence from the Health and Retirement Study Rik-Sharing within Familie: Evidence from the Health and Retirement Study Ş. Nuray Akın and Okana Leukhina December 14, 2014 We report trong empirical upport for the preence of elf-interet-baed rik haring

More information

Auction-Based Resource Allocation for Sharing Cloudlets in Mobile Cloud Computing

Auction-Based Resource Allocation for Sharing Cloudlets in Mobile Cloud Computing 1 Auction-Baed Reource Allocation for Sharing Cloudlet in Mobile Cloud Computing A-Long Jin, Wei Song, Senior Member, IEEE, and Weihua Zhuang, Fellow, IEEE Abtract Driven by pervaive mobile device and

More information

Two Dimensional FEM Simulation of Ultrasonic Wave Propagation in Isotropic Solid Media using COMSOL

Two Dimensional FEM Simulation of Ultrasonic Wave Propagation in Isotropic Solid Media using COMSOL Excerpt from the Proceeding of the COMSO Conference 0 India Two Dimenional FEM Simulation of Ultraonic Wave Propagation in Iotropic Solid Media uing COMSO Bikah Ghoe *, Krihnan Balaubramaniam *, C V Krihnamurthy

More information

Bundled Discounts: Strategic Substitutes or Complements?

Bundled Discounts: Strategic Substitutes or Complements? Bundled Dicount: Strategic Subtitute or Complement? Duarte Brito y Univeridade Nova de Liboa and CEFGE-UE Helder Vaconcelo z Faculdade de Economia, Univeridade do Porto, CEF.UP and CEPR June 2, 24 btract

More information

Software Engineering Management: strategic choices in a new decade

Software Engineering Management: strategic choices in a new decade Software Engineering : trategic choice in a new decade Barbara Farbey & Anthony Finkeltein Univerity College London, Department of Computer Science, Gower St. London WC1E 6BT, UK {b.farbey a.finkeltein}@ucl.ac.uk

More information

Trusted Document Signing based on use of biometric (Face) keys

Trusted Document Signing based on use of biometric (Face) keys Truted Document Signing baed on ue of biometric (Face) Ahmed B. Elmadani Department of Computer Science Faculty of Science Sebha Univerity Sebha Libya www.ebhau.edu.ly elmadan@yahoo.com ABSTRACT An online

More information

Online story scheduling in web advertising

Online story scheduling in web advertising Online tory cheduling in web advertiing Anirban Dagupta Arpita Ghoh Hamid Nazerzadeh Prabhakar Raghavan Abtract We tudy an online job cheduling problem motivated by toryboarding in web advertiing, where

More information

MATLAB/Simulink Based Modelling of Solar Photovoltaic Cell

MATLAB/Simulink Based Modelling of Solar Photovoltaic Cell MATLAB/Simulink Baed Modelling of Solar Photovoltaic Cell Tarak Salmi *, Mounir Bouzguenda **, Adel Gatli **, Ahmed Mamoudi * *Reearch Unit on Renewable Energie and Electric Vehicle, National Engineering

More information

HUMAN CAPITAL AND THE FUTURE OF TRANSITION ECONOMIES * Michael Spagat Royal Holloway, University of London, CEPR and Davidson Institute.

HUMAN CAPITAL AND THE FUTURE OF TRANSITION ECONOMIES * Michael Spagat Royal Holloway, University of London, CEPR and Davidson Institute. HUMAN CAPITAL AND THE FUTURE OF TRANSITION ECONOMIES * By Michael Spagat Royal Holloway, Univerity of London, CEPR and Davidon Intitute Abtract Tranition economie have an initial condition of high human

More information

The Arms Race on American Roads: The Effect of SUV s and Pickup Trucks on Traffic Safety

The Arms Race on American Roads: The Effect of SUV s and Pickup Trucks on Traffic Safety The Arm Race on American Road: The Effect of SUV and Pickup Truck on Traffic Safety Michelle J. White Univerity of California, San Diego, and NBER Abtract Driver have been running an arm race on American

More information

Bio-Plex Analysis Software

Bio-Plex Analysis Software Multiplex Supenion Array Bio-Plex Analyi Software The Leader in Multiplex Immunoaay Analyi Bio-Plex Analyi Software If making ene of your multiplex data i your challenge, then Bio-Plex data analyi oftware

More information

MECH 2110 - Statics & Dynamics

MECH 2110 - Statics & Dynamics Chapter D Problem 3 Solution 1/7/8 1:8 PM MECH 11 - Static & Dynamic Chapter D Problem 3 Solution Page 7, Engineering Mechanic - Dynamic, 4th Edition, Meriam and Kraige Given: Particle moving along a traight

More information

Combining Supervised and Unsupervised Monitoring for Fault Detection in Distributed Computing Systems

Combining Supervised and Unsupervised Monitoring for Fault Detection in Distributed Computing Systems Combining Supervied and Unupervied Monitoring for Fault Detection in Ditributed Computing Sytem Haifeng Chen Guofei Jiang Critian Ungureanu Kenji Yohihira NEC Laboratorie America, Inc. Independence Way

More information

RISK MANAGEMENT POLICY

RISK MANAGEMENT POLICY RISK MANAGEMENT POLICY The practice of foreign exchange (FX) rik management i an area thrut into the potlight due to the market volatility that ha prevailed for ome time. A a conequence, many corporation

More information