A Simple and Efficient Sampling Method for Estimating AP and NDCG
|
|
- Clinton Wilkins
- 7 years ago
- Views:
Transcription
1 A Simple and Efficient Sampling Method for Etimating AP and NDCG Emine Yilmaz Microoft Reearch 7 JJ Thomon Avenue Cambridge CB3 FB, UK Evangelo Kanoula ekanou@cc.neu.edu Javed A. Alam jaa@cc.neu.edu College of Computer and Information Science Northeatern Univerity 36 Huntington Ave, #22 WVH Boton, MA 25 ABSTRACT We conider the problem of large cale retrieval evaluation. Recently two method baed on random ampling were propoed a a olution to the extenive effort required to judge ten of thouand of document. While the firt method propoed by Alam et al. [] i quite accurate and efficient, it i overly complex, making it difficult to be ued by the community, and while the econd method propoed by Yilmaz et al., [4], i relatively imple, it i le efficient than the former ince it employ uniform random ampling from the et of complete judgment. Further, none of thee method provide confidence interval on the etimated value. The contribution of thi paper i threefold: () we derive confidence interval for, (2) we extend to incorporate nonrandom relevance judgment by employing tratified random ampling, hence combining the efficiency of tratification with the implicity of random ampling, (3) we decribe how thi approach can be utilized to etimate ndcg from incomplete judgment. We validate the propoed method uing TREC data and demontrate that thee new method can be ued to incorporate nonrandom ample, a were available in TREC Terabyte track 6. Categorie and Subject Decriptor: H.3 Information Storage and Retrieval; H.3.4 Sytem and Software: Performance Evaluation General Term: Experimentation, Meaurement, Theory Keyword: Evaluation, Sampling, Incomplete Judgment, Average Preciion, ndcg,. INTRODUCTION We conider the problem of large cale retrieval evaluation, in particular, retrieval evaluation with incomplete relevance judgment. We gratefully acknowledge the upport provided by NSF grant IIS and IIS Permiion to make digital or hard copie of all or part of thi work for peronal or claroom ue i granted without fee provided that copie are not made or ditributed for profit or commercial advantage and that copie bear thi notice and the full citation on the firt page. To copy otherwie, to republih, to pot on erver or to reditribute to lit, require prior pecific permiion and/or a fee. SIGIR 8, July 2 24, 28, Singapore. Copyright 28 ACM /8/7...$5.. The mot commonly employed methodology for aeing the quality retrieval ytem i the Cranfield methodology adopted by TREC [7]. The main aumption behind thi methodology i that the relevance judgment are complete, i.e., for each query, all document in the collection relevant to thi query are identified. When the document collection i large, obtaining complete relevance judgment i infeaible due to the need for extenive human effort. Intead, TREC employ depth-k pooling (typically, depth- pooling) to overcome thi burden, combining top k document retrieved by the ubmitted ytem and auming the ret of the document are nonrelevant. While a depth- pool i ignificantly maller than the document collection, it till require extenive judgment effort (e.g. 86,83 judgment for ). Furthermore, even though depth- pool were hown to contain mot of the relevant document [8], the ize of document collection tend to increae and thu thee pool may be inadequate for identifying mot of the relevant document. Recently, evaluation with incomplete relevance judgment ha gained attention a a olution to the problem of extenive judgment effort. New evaluation meaure have been propoed in the literature [, 5,, 2, 4] ince tandard evaluation meaure uch a average preciion are not robut to incomplete judgment [3]. A a olution to thi problem, Buckley and Voorhee [3] propoed bpref, a now commonly ued meaure by information retrieval community. Sakai [2] intead applied traditional meaure to condened lit of document obtained by filtering out all unjudged document from the original ranked lit and howed that thee verion of meaure are actually more robut to incompletene than bpref. Carterette et al. [5] and Moffat et al. [] elect a ubet of document to be judged baed on the benefit document provide in fully ranking ytem or identifying the bet ytem, repectively. Even though the aforementioned approache are all hown to be more robut to incompletene than tandard evaluation meaure, thee method are not guaranteed to compute or etimate the value of tandard evaluation meaure. Hence, the value of meaure obtained by thee method are difficult to interpret. Yilmaz and Alam [4] and Alam et al. [] intead ue random ampling to etimate the actual value of average preciion when relevance judgment are incomplete. Both of thee method are baed on treating incomplete relevance
2 judgment a a ample drawn from the et of complete judgment and uing tatitical method to etimate the actual value of the meaure. Thee method are both hown to () produce unbiaed etimate of average preciion even when relevance judgment are incomplete and (2) be more robut to incomplete relevance judgment than any other meaure uch a bpref [3] or the condened verion of the meaure (referred to a induced AP in the paper)[4]. The meaure propoed by Yilmaz and Alam, [4], became a commonly ued meaure by information retrieval community [2, 3] and wa ued in TREC VID and Terabyte track in 26 [, 4]. A limitation of accrue from the meaure aumption that incomplete relevance judgment are a imple random ample drawn from the et of complete judgment. Typical evaluation meaure give more weight to document retrieved toward the top of a retrieved lit and therefore, a top-heavy ampling trategy would lead to more accurate reult with higher efficiency in term of judgment effort needed. On the other hand, according to the method by Alam et al. [] ample are drawn according to a carefully choen non-uniform ditribution over the document in the depth- pool. Even though thi method i more efficient in term of judgment effort than, it i very complex both in conception and implementation and therefore le applicable. Furthermore, although average preciion etimator a propoed by both of the aforementioned method are unbiaed in expectation, in practice, when calculated uing a ingle ample of relevance judgment, may vary in value. Thi neceitate the derivation and ue of confidence interval around the etimated value in order to allow confident concluion regarding the actual value of average preciion and thu the ranking of retrieval ytem. In thi paper, we mainly focu on inferred average preciion. Firt, we derive confidence interval for the meaure and validate them uing TREC data. We how that along with the correponding confidence interval can allow reearcher to reach confident concluion about actual average preciion, even when relevance judgment are incomplete. We then focu on the efficiency of the meaure. We employ a tratified random ampling methodology and extend the meaure to incorporate relevance judgment created according to any uch ampling ditribution. Thi extended combine the implicity of random ampling with the efficiency of tratification and thu it i imple and eay to compute while, at the ame time, it i much more efficient than in term of reducing the judgment effort. We further claim that the ame methodology can be applied to other evaluation meaure and demontrate how ndcg (a commonly ued meaure that incorporate graded relevance judgment [9]) can be etimated uing incomplete relevance judgment. 2. CONFIDENCE INTERVALS FOR INFAP The inferred average preciion, by tatitical contruction, i an unbiaed etimator of average preciion and thu it i deigned to be exactly equal to average preciion in expectation. However in practice, it may be low or high due to the nature of ampling (epecially when the ubet of document whoe binary relevance i available i mall). In other word, there i variability in the value of becaue different ample from the collection of document give rie to different value of. The amount of the variability in i meaured by it variance. Before computing the variance of let reviit the random experiment whoe expectation i average preciion [4] and identify all ource of variability in the outcome of thi random experiment. Given a ranked lit of document with repect to a given topic:. Select a relevant document at random and let the rank of thi relevant document in lit be k. 2. Select a rank, j, at random from the et {,..., k}. 3. Output the binary relevance of the document at rank j. In expectation, tep (2) and (3) effectively compute the preciion at a relevant document and in combination, tep () compute the average of thee preciion. The aforementioned experiment can be realized a a twotage ampling. At the firt tage tep () a ample of cut-off level at relevant document i elected. The value i computed a an average of the etimated preciion value at the ampled cut-off level. Even if we aume that thee preciion value are the actual preciion value, varie becaue different ample of cut-off level will reult in different value of. Therefore, computing uing preciion value only at a ubet of cut-off level introduce the firt component of variability. Let rel be the et of the judged relevant document of ize r. Thi firt variance component can be etimated a var. comp. = ( p) 2 /r where p % i the ampling percentage and 2 the variance among the preciion value at the judged relevant document P calculated a 2 = k rel (d P C k ) /r. At the econd tage tep (2) for each one of the elected cut-off level, a ample of document above that cutoff level document i ued to etimate the correponding to the cut-off preciion value. Therefore, even for a given ample of cut-off level, ha variability becaue different ample of document give rie to different value of preciion and thu different value of. Hence, computing the preciion at ome cut-off uing only a ubet of the document above that cut-off introduce a econd component of variability. Auming that preciion at different cut-off level are independent from each other, thi econd variance component can be etimated a, X var. comp. 2 = var[ d P C k ] /r 2 k rel where var[ P d C k ] i the variance of the etimated preciion at cut-off k. According to the Law of Total Variance, the total variance of can be computed a the um of the two aforementioned variance component; hence, var[] = ( p) 2 r + P k rel var[d P C k ] r 2 The complete formula of variance along with the derivation can be found at the Appendix
3 When evaluating retrieval ytem, the average of value acro all topic (mean ) i employed. The variance of the mean can be computed a a function of the variance of a var[mean ] = X var[]/(# of querie) 2 According to the Central Limit Theorem one can aign 95% confidence interval to mean a a function of it variance. A 95% confidence interval centered at the mean intimate that with 95% confidence the actual value of MAP i within thi interval. We ued,9 and data to validate the derived variance of the mean when relevance judgment are incomplete. We imulated the effect of incomplete relevance judgment a in [4]. For each TREC, we formed incomplete judgment et by ampling from the entire depth- pool over all ubmitted run. Thi i done by electing p% of the complete judgment et uniformly at random, where p {, 2, 3}. The reult of our experiment led to identical concluion over all TREC dataet and therefore, due to pace limitation, we report only reult for. Figure illutrate the mean value computed from a ingle random ample of document per topic for each run againt the actual MAP value for p {, 2, 3} for. The 95% confidence interval are depicted a error bar around the mean value. A one can oberve, the greatet majority of the confidence interval interect the 45 o dahed line indicating that the greatet majority of the confidence interval cover the actual MAP value. Furthermore, we computed the mean value and the correponding confidence interval for different ampling trial over data and we accumulated the deviation of the computed mean value from the actual MAP value in term of tandard deviation. Thi way we generated a Cumulative Ditribution Function of divergence of mean value per ytem. According to the Central Limit Theorem each of thee CDF hould match the CDF of the Normal Ditribution. We performed a Kolmogorov- Smirnov tet of fitne and for 9% of the ytem the hypothei that the two CDF match could not be rejected (α =.5) which validate our derived theoretical reult. 3. INFERRED AP ON NONRANDOM JUDGMENTS In the previou ection we derived confidence interval for in a etup where document to be judged were a random ubet of the entire document collection. Confidence interval can be further reduced (i.e. the accuracy of the etimator can be improved) by utilizing a top-heavy ampling trategy. In thi ection we conider a etup where relevance judgment are not a random ubet of complete judgment and how how can be extended to produce unbiaed etimate of average preciion in uch a etup. We denote the extended meaure a x. Similar to the paradigm, conider the cae where we would like to evaluate the quality of retrieval ytem with repect to a complete pool and aume that relevant judgment are incomplete. Further aume that the et of available judgment are contructed by diving the complete collection of document into dijoint contiguou ubet (trata) and then randomly electing (ampling) ome document from each tratum to be judged. The ampling within each tratum i performed independently, therefore, the ampling percentage can be choen to be different for each tratum. For intance, one could chooe to plit the collection of document into two trata (baed on where they appear in the output of earch engine), and ample 9% of the document from the firt tratum and 3% of the document from the econd tratum. In effect, one could think a large variety of ampling trategie in term of thi multi-trata trategy. For example, the ampling trategy propoed by Alam et al. [] can be thought a each tratum containing a ingle document, with different ampling probabilitie aigned to different trata. Let AP d be the random variable correponding to the etimated average preciion of a ytem. Now conider the firt tep of the random experiment whoe expectation correpond to average preciion, i.e. picking a relevant document at random. Note that in the above etup, thi relevant document could fall into any one of the different trata. Since the et of document contained in the trata are dijoint, by definition of conditional expectation, one can write E[ AP d ] a: E[ AP d X ] = P E [ AP d ] Strata where P correpond to the probability of picking the relevant document from tratum and E [ AP d ] correpond to the expected value of average preciion given that the relevant document wa picked from tratum. Let R Q be the total number of relevant document in the complete judgment et and R be the total number of relevant document in tratum if we were to have all complete relevance judgment. Then, ince electing document from different trata i independent for each tratum, the probability of picking a relevant document from tratum i, P = R /R Q. Computing the actual value of R Q and R i not poible, ince the complete et of judgment i not available. However, we can etimate their value uing the incomplete relevance judgment. Let r be the number of ampled relevant document and n be the total number of ampled document from tratum. Furthermore, let N be the total number of document in tratum. Since the n document were ampled uniformly from tratum, the etimated number of relevant document within tratum, ˆR, can be computed a ˆR = (r /n ) N. Then the number of relevant document in query Q can be etimated a the um of thee etimate over all trata, i.e. ˆRQ = P ˆR. Given thee etimate, the probability of picking a relevant document from tratum can be etimated by, ˆP = ˆR / ˆR Q. Now, we need to compute the expected value of etimated average preciion, E [ AP d ], if we were to pick a relevant document at random from tratum. Since the incomplete relevance judgment within each tratum i a uniform random ubet of the judgment in that tratum, the induced ditribution over relevant document within each tratum i alo uniform, a deired. Therefore, the probability of picking any relevant document within thi tratum i equal. Hence, the expected etimated average preciion value within each tratum, E [ AP d ], can be computed a the average of the preciion at judged (ampled) relevant document within that tratum.
4 .6, Sample Percentage =%.6, Sample Percentage =2%, Sample Percentage =3% Inferred MAP.3.2 Inferred MAP.3.2 Inferred MAP Actual MAP Actual MAP Actual MAP Figure : TREC-8 mean inferred AP along with etimated confidence interval when relevance judgment are generated by ampling, 2 and 3 % of the depth- pool veru the mean actual AP. Now conider computing the expected preciion at a relevant document at rank k, which correpond to the expected outcome of picking a document at or above rank k and outputting the binary relevance of the document at thi rank (tep 2 and 3 of the random experiment). When picking a document at random at or above rank k and outputting the binary relevance of that document, one of the following two cae may occur. With probability /k, we pick the current document, and ince thi document i by definition relevant the outcome i. With probability (k )/k we pick a document above the current document, in which cae we need to calculate the expected preciion (or expected binary relevance) with repect to the document above rank k. Thu, E[ d P C k ] = k + k k E[d P C above k ] Let N k be the total number of document above rank k that belong tratum, n k be the total number of judged (ampled) document above rank k that belong to tratum and r k be the total number of judged (ampled) relevant document above rank k that alo belong to tratum. When computing the expected preciion within the (k ) document above rank k, with probability N k /(k ) we pick a document from tratum. Therefore, the expected preciion above rank k can be written a: E[prec above k] = X N k k E[d P C above k ] where E [ P d C above k ] i the expected preciion above rank k within tratum. Since we have a uniform ample of judged document from tratum, we can ue thee ampled document to etimate the expected preciion within tratum. Since the incomplete relevance judgment from each tratum i obtained by uniform random ampling, thi expected preciion can be computed a r k /n k. Note that in computing the expected preciion in tratum, we may face the problem of not having ampled any document from thi tratum that are above the current relevant document at rank k. Adapting the ame idea ued in, we employ Lindtone moothing [6] to avoid thi problem. Therefore, expected preciion above rank k can be computed a: E[ d P C above k ] = X N k k r k n k + ɛ + 2ɛ It i eay to ee that when complete judgment are available, x i exactly equal to average preciion (ignoring the moothing effect). Further, note that i a particular intantiation of thi formula with a ingle tratum ued. Overall, the advantage and real power of the decribed tratified random ampling and the derived AP etimator, x, i the fact that it combine the effectivene of the ampling method propoed by Alam et al. [] by employing tratification of the document and thu better utilization of the judgment effort with the implicity of by employing random ampling within each tratum. 3. Inferred AP in TREC Terabyte A mentioned, x can be ued with a large variety of ampling trategy. In thi ection, we focu on the ampling trategy ued in TREC Terabyte 26 [4] and we how that () x i highly effective at etimating average preciion and (2) it better utilize the judgment effort compared to. Firt, let briefly conider the ampling trategy ued in TREC Terabyte 26. In thi track, three different et of relevance judgment were formed, with only two of them being ued for evaluation purpoe. Out of thee two et, the firt et of judgment, contructed by the traditional depth-5 pooling trategy, wa ued to obtain a rough idea of the ytem average preciion. The econd et of judgment wa contructed uing random ampling in uch a way that there are more document judged from topic that are more likely to have retrieved more relevant document. Since, in Terabyte track, the ize of the document collection i very large, the ytem may continue retrieving relevant document even at high rank (deeper in the lit). Thi et of judgment wa created to obtain an etimate of average preciion if complete judgment were preent. To etimate average preciion, wa ued a the evaluation meaure. Since, by deign, aume that the et of relevance judgment i a random ubet of complete judgment, even though the entire depth-5 pool wa judged, wa computed only uing the random ample of judgment (econd et) without utilizing judgment from the depth-5 pool. Therefore, many relevance judgment were not ued even though they were available. Note that x can eaily handle thi etup and it could be ued to utilize all the judgment, obtaining better etimate of average preciion. To tet how x compare with we imulate the ampling trategy ued in TREC Terabyte 6 on data from
5 . The TREC Terabyte data wa not ued due to the fact that in TREC Terabyte the actual value of average preciion i not known ince complete judgment are not available. To imulate the etup ued in TREC Terabyte, we firt form different depth-k pool where k {, 2, 3, 4, 5,, 2, 3, 4, 5} and obtain judgment for all document in each one of thee pool. Then, for each value of k, we compute the total number of document that are in the depth-k pool and we randomly ample equal number of document from the complete judgment et 2 excluding the depth-k pool. After forming thee two et of judgment (depth-k and random) we combine them and compute x on thee combined judgment. Thi etup exactly correpond to a ampling trategy where complete judgment are divided into two trata and judgment are formed by uniformly and independently ampling within each tratum. Note that in TREC, there are ome ytem that were ubmitted but that did not contribute to the pool. To further evaluate the quality of our etimator in term of their robutne for evaluating the quality of uneen ytem (ytem that did not contribute to the pool), when we form the incomplete relevance judgment, we only conider the ytem that contribute to the pool but we compute the x etimate for all ubmitted ytem. Figure 2 demontrate how x computed uing judgment generated by combining (left) depth-, (middle) depth- 5 and (right) depth- pool with equal number of randomly ampled judgment compare with the actual AP. Each of thee depth correpond to judging 23.%, 2.7% and 3.5% of the entire pool, repectively. The plot report the RMS error (how accurate are the etimated value?), the Kendall τ value (how accurate are the etimated ranking of ytem?) and the linear correlation coefficient, ρ, (how well do the etimated value fit in a traight line compared to the actual value?). The dot ign in the figure refer to the ytem that were ued to create the original pool and the plu ign refer to the ytem that did not contribute to the pool. The reult illutrated in thee plot reinforce our claim that x i an unbiaed etimator of average preciion. Furthermore, it can be een that the meaure can reliably be ued to evaluate the quality of ytem that were not ued to create the initial ample, hence the meaure i robut to evaluating the quality of uneen ytem. Figure 3 illutrate how x computed on a non-random judgment et compare with computed on a random judgment et for variou level of incompletene. In a imilar manner to the experimental etup of the original work, for each value of k, we generated ten different ample trial according to the procedure decribed in the previou paragraph, and for each one of the ten trial we computed the x for all ytem. Then, all three tatitic were computed for each one of the trial and the average of thee tatitic over all ten trial were reported for different level of judgment incompletene. Uing the ame procedure, we alo created ten different ample trial where the ample were generated by merely randomly ampling the judgment 2 Throughout thi paper, we aume that the complete judgment et correpond to the depth- pool a the judgment we have are formed uing depth- pool and auming the remaining document are nonrelevant. et and the value were computed on them. For comparion purpoe, to how how the original verion of behave when thi randomne aumption i violated, we alo include run on the ame judgment et a extended (marked a depth+random judgment in the Figure). It can be een that for all level of incompletene, in term of all three tatitic, x i much more accurate in etimating average preciion than the other two meaure. We further compared x to the ampling method propoed by Alam et al. []. The robutne of x to incomplete relevance judgment i comparable to (and in ome cae even better than) thi method. (Thee reult were omitted due to pace limitation.) 4. ESTIMATION OF NDCG WITH INCOMPLETE JUDGMENTS There are different verion of the ndcg metric depending on the dicount factor and the gain aociated with relevance grade, etc. In thi paper, we adopt the verion of ndcg in trec_eval. Let I denote a relevance grade and gain(i) the gain aociated with I. Alo, let g, g 2,... g Z be the gain value aociated with the Z document retrieved by a ytem in repone to a query q, uch a g i = gain(i) if the relevance grade of the document in rank i i I. Then, the ndcg value for thi ytem can be computed a, ndcg = DCG DCG I where DCG = ZX g i/ lg(i + ) and DCG I denote the DCG value for an ideal ranked lit for query q. The etimation of ndcg with incomplete judgment can be divided into two part: () Etimating DCG I and (2) Etimating DCG. Then, the DCG and the DCG I value can be replaced by their etimate to obtain the etimated ndcg value Etimating DCG I The normalization factor, DCG I, for a query q can be defined a the maximum poible DCG value over that query. Hence, the etimation of DCG I can be derived in a two-tep proce: () For each relevance grade I uch a gain(i) >, etimate the number of document with that relevance grade; (2) Calculate the DCG value of an optimal lit by auming that in an optimal lit the etimated number of document would be orted (in decending order) by their relevance grade. Uing the ampling trategy decribed in the previou ection, uppoe incomplete relevance judgment were created by diving the complete pool into dijoint et (trata) and randomly picking (ampling) document from each tratum to be judged, poibly with different probability for each tratum. 3 Note that thi aume that E[nDCG] = E[DCG]/E[DCG I], i.e., that DCG I and DCG are independent of each other, which i not necearily the cae. Thi aumption may reult in a mall bia and better etimate of ndcg can be obtained by conidering thi dependence. However, for the ake of implicity, throughout thi paper, we will aume that thee term are independent. i=
6 xinf map v. actual map, 2.7% judgment xinf map v. actual map, 3.5% judgment.5 Pool Sy RMS =.26, τ =.9345, ρ =.9963 Non Pool Sy RMS =.33, τ =.8678, ρ = Pool Sy RMS =.26, τ =.8699, ρ =.982 Non Pool Sy RMS =.53, τ =.848, ρ =.987 extended inferred map extended inferred map actual map actual map Figure 2: TREC-8 mean x when relevance judgment are generated according to depth-, depth-5 and depth- pooling combined with equivalent number of randomly ampled judgment veru mean actual AP..25 InfAP depth + random judgment extended InfAP random judgment Kendall τ Linear correlation coefficient (ρ) RMS error InfAP depth + random judgment.95 InfAP depth + random judgment.5.7 InfAP random judgment InfAP random judgment Figure 3: TREC-8 change in Kendall τ, linear correlation coefficient (ρ), and RMS error of x and a the judgment et are reduced when half of the judgment are generated according to depth pooling and the other half i a random ubet of complete judgment and of inferred AP when the judgment are a random ubet of complete judgment. For each tratum, let r (I) be the number of ampled document with relevance grade I, let n be the total number of document ampled from trata and N be the total number of document that fall in trata. Since the n document are ampled uniformly from trata, the etimated number of document with relevance grade I within thi trata can be computed a ˆR (I) = r(i) N n Then, the expected number of document with relevance grade I within the complete pool can be computed a ˆR(I) = X ˆR (I) Once thee etimate are obtained, one can etimate DCG I. 4.2 Etimating DCG Given Z document retrieved by a earch engine with relevance gain g i for the document at rank i, for each rank g i, define a new variable x i uch a x i = Z i. Then, lg(i+) DCG can be written a the output of the following random experiment:. Pick a document at random from the output of the earch engine, let the rank of thi document be i. 2. Output the value of x i. It i eay to ee that if we have the relevance judgment for all Z document, the expected value of thi random experiment i exactly equal to DCG. Now conider etimating the outcome of thi random experiment when relevance judgment are incomplete. Conider the firt tep of the random experiment, i.e. picking a document at random. Let Z be the number of document in the output of a ytem that fall in tratum. When picking a document at random, with probability Z /Z, we pick a document from tratum. Therefore, the expected value of the above random experiment can be written a: E[DCG] = X Z Z E[xi document at rank i ] Now conider the econd tep of the random experiment, computing the expected value of x i given that the document at rank i fall in trata. Let ampled be the et of ampled document from trata and n be the number of document ampled from thi trata. Since document within tratum are uniformly ampled, the expected value of x i can be computed a E[x i document at rank i ] = X n j ampled x j Once E[DCG I] and E[DCG] are computed, infndcg can then be computed a infndcg = E[DCG]/E[DCG I]. 5. OVERALL RESULTS Until now, we have hown that uing a imilar ampling trategy a the one ued in TREC Terabyte 6 (complete
7 judgment divided into 2 different trata), x i highly accurate. In thi ection, we how that () thi claim i conitent over different TREC for both x and infndcg and that (2) the two meaure can be ued with the complete judgment divided into more than two trata. In order to check (2), we ue a different ampling trategy than the one in Terabyte; we divide the complete judgment et (auming depth- pool i the complete judgment et) into 4 different trata. The firt tratum i the regular depthk pool, fully judged. Intead of randomly ampling equal to the depth-k pool number of judgment from the remainder of the collection, we now divide the ret of the document into three other trata and ditribute the remaining judgment with a ratio of 3:.5: (judge 55% of the document in the top depth tratum, 27% of the document in the middle depth tratum and 8% in the lowet depth tratum). Thi way, more weight i given to judging document retrieved toward the top of the ranked lit of the earch engine. Note, however, that a the number of trata increae, there value of the etimate may lightly deviate from the actual value ince the effect of moothing alo increae (moothing i needed for each tratum). Figure 4 how the quality of x and infndcg (referred a to avoid confuion) computed on thee ample according to Kendall τ and RMS Error tatitic, for, 9 and. For comparion purpoe, the plot alo contain and ndcg (the tandard formula computed on random judgment, auming unjudged document are nonrelevant). Looking at all plot, it can be een that according to both tatitic, uing the ame number of judgment, the extended (x) and infndcg conitently outperform and ndcg on random judgment, repectively. The high RMS error of ndcg on random judgment i due to the fact that ndcg i computed on thee judgment a it i, without aiming at etimating the value of the meaure. 6. CONCLUSIONS In thi work, we extended inferred AP in two different way. Firt, we derived confidence interval for to capture the variability in value. Employing confidence interval enable comparion and eventually ranking of ytem according to their quality meaured by AP with high confidence. Second, we utilized a tratified random ampling trategy to elect document to be judged and extended to handle the non-random ample of judgment. We applied the ame methodology for etimating ndcg in the preence of incomplete non-random judgment. Stratified random ampling combine the effectivene of tratification and thu better utilization of the relevance judgment with the implicity of random ampling. We howed that x and infndcg are more accurate than and ndcg on equal number of random ample. Note that the ampling trategy (i.e. the number of trata, the ize of each tratum and the ampling percentage from each tratum) ued here i rather arbitrary. The confidence interval a decribed in the firt part of thi paper could be ued a an objective function to determine an optimal ampling trategy. The ampling trategy i highly important for the quality of the etimate and identifying an optimal trategy i a point of future reearch. Furthermore, confidence interval a a function of the ample ize could be ued to determine the appropriate number of document to be judged for an accurate MAP etimation which i a point we plan to invetigate. 7. REFERENCES [] J. A. Alam, V. Pavlu, and E. Yilmaz. A tatitical method for ytem evaluation uing incomplete judgment. In Proceeding of the 29th annual international ACM SIGIR conference on Reearch and development in information retrieval, page ACM Pre, Augut 26. [2] T. Bompada, C.-C. Chang, J. Chen, R. Kumar, and R. Shenoy. On the robutne of relevance meaure with incomplete judgment. In SIGIR 7: Proceeding of the 3th annual international ACM SIGIR conference on Reearch and development in information retrieval, page , New York, NY, USA, 27. ACM Pre. [3] C. Buckley and E. M. Voorhee. Retrieval evaluation with incomplete information. In Proceeding of the 27th Annual International ACM SIGIR Conference on Reearch and Development in Information Retrieval, page 25 32, 24. [4] S. Buttcher, C. Clarke, and I. Soboroff. The TREC 26 terabyte track. In Proceeding of the Fifteenth Text REtrieval Conference (TREC 26), 26. [5] B. Carterette, J. Allan, and R. Sitaraman. Minimal tet collection for retrieval evaluation. In Proceeding of the 29th Annual International ACM SIGIR Conference on Reearch and Development in Information Retrieval, page , 26. [6] S. F. Chen and J. Goodman. An empirical tudy of moothing technique for language modeling. In Proceeding of the Thirty-Fourth Annual Meeting of the Aociation for Computational Linguitic, page 3 38, San Francico, 996. Morgan Kaufmann Publiher. [7] C. Cleverdon. The cranfield tet on index language device. Reading in information retrieval, page 47 59, 997. [8] D. Harman. Overview of the third text REtreival conference (TREC-3). In D. Harman, editor, Overview of the Third Text REtrieval Conference (TREC-3), page 9. U.S. Government Printing Office, Apr [9] K. Järvelin and J. Kekäläinen. Cumulated gain-baed evaluation of ir technique. ACM Tranaction on Information Sytem, 2(4): , 22. [] W. Kraaij, P. Over, and A. Smeaton. TRECVID 26 - an introduction. In TREC Video Retrieval Evaluation Online Proceeding, 26. [] A. Moffat, W. Webber, and J. Zobel. Strategic ytem comparion via targeted relevance judgment. In SIGIR 7: Proceeding of the 3th annual international ACM SIGIR conference on Reearch and development in information retrieval, page , New York, NY, USA, 27. ACM. [2] T. Sakai. Alternative to bpref. In SIGIR 7: Proceeding of the 3th annual international ACM SIGIR conference on Reearch and development in information retrieval, page 7 78, New York, NY, USA, 27. ACM Pre. [3] I. Soboroff. A comparion of pooled and ampled relevance judgment. In SIGIR 7: Proceeding of the 3th annual international ACM SIGIR conference on Reearch and development in information retrieval, page , New York, NY, USA, 27. ACM Pre. [4] E. Yilmaz and J. A. Alam. Etimating average preciion with incomplete and imperfect judgment. In Proceeding of the Fifteenth ACM International Conference on Information and Knowledge Management. ACM Pre, November 26. APPENDIX Let d be a ample of cut-off level at relevant document. According to the Law of Total Variance, the variance in can be calculated a, var[] = var[e[ d ]] + E[var[ d ]] Let conider the firt term of the right-hand ide of the above equation, which correpond to the variance due to ampling cut-off level.
8 5432 TREC 9 TREC Kendall τ.85 Kendall τ.9.85 Kendall τ percentage of pool judged percentage of pool judged percentage of pool judged TREC TREC.3.25 RMS error RMS error.5. RMS error Figure 4: Comparion of extended inferred map, (extended) mean inferred ndcg, inferred map and mean ndcg on random judgment, uing Kendall τ (firt row) and RMS error (lat row) for, 9 and. Let r the number of relevant document in d. Then, the conditional expectation of i, E[ d ] = X E[ P r d C k d ] = X P C k r k d k d where P d C k and P C k denote the etimated and actual preciion at cut-off k, repectively. Thu, 2 3 var[e[ d ]] = var 4 X P C k 5 = ( p) σ2 r r k d where p% i the ampling percentage of document from the entire depth- pool and σ 2 i the actual variance among the preciion value at all cut-off of relevant document and P it can be etimated by, (d P C k d k ) 2 /(r ). Now, let conider the econd term of the right-hand ide of the equation deduced by the Law of Total Variance, that i the variance due to ampling document above a cut-off level in order to etimate the preciion at that cut-off level, var[ d ] = var 4 X P d Ck 5 = r r var 4 X dp C 2 k 5 k d k d Conidering d P C k independent from each other If we conider preciion at different cut-off level independent from each other the variance of for a given et of ampled cut-off level depend on the ummation of the preciion variance at each individual cut-off level, var[ d ] = X var[ P r d C 2 k d ] k d The preciion at cut-off i alway and therefore the variance i. Moreover, the preciion at relevant document not in the retrieved lit i alway aumed to be and therefore, the variance at thoe cut-off level i alo. In all other cae dp C k i calculated a, P d C k = /k + ((k )/k) dp C above k and therefore, «2 var[ P d k C k d ] = var[ P k d C above k ] Let r k and n k be the number of relevant document and total number of document ampled above cut-off k, repectively and let d k be the number of document in the depth- pool above cut-off k. The preciion above cut-off k i etimated by 4, P d C k = d k rk k n k, which follow a hypergeometric ditribution and it variance can be calculated a, var[ P d p( p) C k d ] = n k «n «k d k By conidering the expected value of var[ d ] over all ample of cut-off level we get, P k E[var[ d ]] = d var[ P d C k d ] r 2 Conidering d P C k dependent to each other If we do not conider preciion at different cut-off level independent from each other the covariance between preciion can be calculated a, cov[ d P C k, d P C m] = k m var[d P C k ] where k < m 4 For implicity reaon we ignore the effect of moothing that i introduced in the formula of. Smoothing wa conidered in all experiment ran and it wa oberved that the effect of moothing in variance i negligible.
A technical guide to 2014 key stage 2 to key stage 4 value added measures
A technical guide to 2014 key tage 2 to key tage 4 value added meaure CONTENTS Introduction: PAGE NO. What i value added? 2 Change to value added methodology in 2014 4 Interpretation: Interpreting chool
More informationAssessing the Discriminatory Power of Credit Scores
Aeing the Dicriminatory Power of Credit Score Holger Kraft 1, Gerald Kroiandt 1, Marlene Müller 1,2 1 Fraunhofer Intitut für Techno- und Wirtchaftmathematik (ITWM) Gottlieb-Daimler-Str. 49, 67663 Kaierlautern,
More informationDISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS
DISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS Chritopher V. Kopek Department of Computer Science Wake Foret Univerity Winton-Salem, NC, 2709 Email: kopekcv@gmail.com
More informationDISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS. G. Chapman J. Cleese E. Idle
DISTRIBUTED DATA PARALLEL TECHNIQUES FOR CONTENT-MATCHING INTRUSION DETECTION SYSTEMS G. Chapman J. Cleee E. Idle ABSTRACT Content matching i a neceary component of any ignature-baed network Intruion Detection
More informationOptical Illusion. Sara Bolouki, Roger Grosse, Honglak Lee, Andrew Ng
Optical Illuion Sara Bolouki, Roger Groe, Honglak Lee, Andrew Ng. Introduction The goal of thi proect i to explain ome of the illuory phenomena uing pare coding and whitening model. Intead of the pare
More informationT-test for dependent Samples. Difference Scores. The t Test for Dependent Samples. The t Test for Dependent Samples. s D
The t Tet for ependent Sample T-tet for dependent Sample (ak.a., Paired ample t-tet, Correlated Group eign, Within- Subject eign, Repeated Meaure,.. Repeated-Meaure eign When you have two et of core from
More informationQueueing systems with scheduled arrivals, i.e., appointment systems, are typical for frontal service systems,
MANAGEMENT SCIENCE Vol. 54, No. 3, March 28, pp. 565 572 in 25-199 ein 1526-551 8 543 565 inform doi 1.1287/mnc.17.82 28 INFORMS Scheduling Arrival to Queue: A Single-Server Model with No-Show INFORMS
More information1 Introduction. Reza Shokri* Privacy Games: Optimal User-Centric Data Obfuscation
Proceeding on Privacy Enhancing Technologie 2015; 2015 (2):1 17 Reza Shokri* Privacy Game: Optimal Uer-Centric Data Obfucation Abtract: Conider uer who hare their data (e.g., location) with an untruted
More informationMulti-Objective Optimization for Sponsored Search
Multi-Objective Optimization for Sponored Search Yilei Wang 1,*, Bingzheng Wei 2, Jun Yan 2, Zheng Chen 2, Qiao Du 2,3 1 Yuanpei College Peking Univerity Beijing, China, 100871 (+86)15120078719 wangyileipku@gmail.com
More informationA note on profit maximization and monotonicity for inbound call centers
A note on profit maximization and monotonicity for inbound call center Ger Koole & Aue Pot Department of Mathematic, Vrije Univeriteit Amterdam, The Netherland 23rd December 2005 Abtract We conider an
More informationA Note on Profit Maximization and Monotonicity for Inbound Call Centers
OPERATIONS RESEARCH Vol. 59, No. 5, September October 2011, pp. 1304 1308 in 0030-364X ein 1526-5463 11 5905 1304 http://dx.doi.org/10.1287/opre.1110.0990 2011 INFORMS TECHNICAL NOTE INFORMS hold copyright
More informationA Spam Message Filtering Method: focus on run time
, pp.29-33 http://dx.doi.org/10.14257/atl.2014.76.08 A Spam Meage Filtering Method: focu on run time Sin-Eon Kim 1, Jung-Tae Jo 2, Sang-Hyun Choi 3 1 Department of Information Security Management 2 Department
More informationTHE IMPACT OF MULTIFACTORIAL GENETIC DISORDERS ON CRITICAL ILLNESS INSURANCE: A SIMULATION STUDY BASED ON UK BIOBANK ABSTRACT KEYWORDS
THE IMPACT OF MULTIFACTORIAL GENETIC DISORDERS ON CRITICAL ILLNESS INSURANCE: A SIMULATION STUDY BASED ON UK BIOBANK BY ANGUS MACDONALD, DELME PRITCHARD AND PRADIP TAPADAR ABSTRACT The UK Biobank project
More informationRedesigning Ratings: Assessing the Discriminatory Power of Credit Scores under Censoring
Redeigning Rating: Aeing the Dicriminatory Power of Credit Score under Cenoring Holger Kraft, Gerald Kroiandt, Marlene Müller Fraunhofer Intitut für Techno- und Wirtchaftmathematik (ITWM) Thi verion: June
More informationPartial optimal labeling search for a NP-hard subclass of (max,+) problems
Partial optimal labeling earch for a NP-hard ubcla of (max,+) problem Ivan Kovtun International Reearch and Training Center of Information Technologie and Sytem, Kiev, Uraine, ovtun@image.iev.ua Dreden
More informationINFORMATION Technology (IT) infrastructure management
IEEE TRANSACTIONS ON CLOUD COMPUTING, VOL. 2, NO. 1, MAY 214 1 Buine-Driven Long-term Capacity Planning for SaaS Application David Candeia, Ricardo Araújo Santo and Raquel Lope Abtract Capacity Planning
More informationMorningstar Fixed Income Style Box TM Methodology
Morningtar Fixed Income Style Box TM Methodology Morningtar Methodology Paper Augut 3, 00 00 Morningtar, Inc. All right reerved. The information in thi document i the property of Morningtar, Inc. Reproduction
More informationUnit 11 Using Linear Regression to Describe Relationships
Unit 11 Uing Linear Regreion to Decribe Relationhip Objective: To obtain and interpret the lope and intercept of the leat quare line for predicting a quantitative repone variable from a quantitative explanatory
More informationMixed Method of Model Reduction for Uncertain Systems
SERBIAN JOURNAL OF ELECTRICAL ENGINEERING Vol 4 No June Mixed Method of Model Reduction for Uncertain Sytem N Selvaganean Abtract: A mixed method for reducing a higher order uncertain ytem to a table reduced
More informationA Resolution Approach to a Hierarchical Multiobjective Routing Model for MPLS Networks
A Reolution Approach to a Hierarchical Multiobjective Routing Model for MPLS Networ Joé Craveirinha a,c, Rita Girão-Silva a,c, João Clímaco b,c, Lúcia Martin a,c a b c DEEC-FCTUC FEUC INESC-Coimbra International
More informationTRADING rules are widely used in financial market as
Complex Stock Trading Strategy Baed on Particle Swarm Optimization Fei Wang, Philip L.H. Yu and David W. Cheung Abtract Trading rule have been utilized in the tock market to make profit for more than a
More informationMeasuring the Ability of Score Distributions to Model Relevance
Meauring the Ability of Score Ditribution to Model Relevance Ronan Cummin Department of Information Technology National Univerity of Ireland, Galway ronan.cummin@nuigalway.ie Abtract. Modelling the core
More informationScheduling of Jobs and Maintenance Activities on Parallel Machines
Scheduling of Job and Maintenance Activitie on Parallel Machine Chung-Yee Lee* Department of Indutrial Engineering Texa A&M Univerity College Station, TX 77843-3131 cylee@ac.tamu.edu Zhi-Long Chen** Department
More informationTap Into Smartphone Demand: Mobile-izing Enterprise Websites by Using Flexible, Open Source Platforms
Tap Into Smartphone Demand: Mobile-izing Enterprie Webite by Uing Flexible, Open Source Platform acquia.com 888.922.7842 1.781.238.8600 25 Corporate Drive, Burlington, MA 01803 Tap Into Smartphone Demand:
More information1) Assume that the sample is an SRS. The problem state that the subjects were randomly selected.
12.1 Homework for t Hypothei Tet 1) Below are the etimate of the daily intake of calcium in milligram for 38 randomly elected women between the age of 18 and 24 year who agreed to participate in a tudy
More informationREDUCTION OF TOTAL SUPPLY CHAIN CYCLE TIME IN INTERNAL BUSINESS PROCESS OF REAMER USING DOE AND TAGUCHI METHODOLOGY. Abstract. 1.
International Journal of Advanced Technology & Engineering Reearch (IJATER) REDUCTION OF TOTAL SUPPLY CHAIN CYCLE TIME IN INTERNAL BUSINESS PROCESS OF REAMER USING DOE AND Abtract TAGUCHI METHODOLOGY Mr.
More informationTIME SERIES ANALYSIS AND TRENDS BY USING SPSS PROGRAMME
TIME SERIES ANALYSIS AND TRENDS BY USING SPSS PROGRAMME RADMILA KOCURKOVÁ Sileian Univerity in Opava School of Buine Adminitration in Karviná Department of Mathematical Method in Economic Czech Republic
More informationName: SID: Instructions
CS168 Fall 2014 Homework 1 Aigned: Wedneday, 10 September 2014 Due: Monday, 22 September 2014 Name: SID: Dicuion Section (Day/Time): Intruction - Submit thi homework uing Pandagrader/GradeScope(http://www.gradecope.com/
More informationPerformance of Multiple TFRC in Heterogeneous Wireless Networks
Performance of Multiple TFRC in Heterogeneou Wirele Network 1 Hyeon-Jin Jeong, 2 Seong-Sik Choi 1, Firt Author Computer Engineering Department, Incheon National Univerity, oaihjj@incheon.ac.kr *2,Correponding
More informationReview of Multiple Regression Richard Williams, University of Notre Dame, http://www3.nd.edu/~rwilliam/ Last revised January 13, 2015
Review of Multiple Regreion Richard William, Univerity of Notre Dame, http://www3.nd.edu/~rwilliam/ Lat revied January 13, 015 Aumption about prior nowledge. Thi handout attempt to ummarize and yntheize
More informationProgress 8 measure in 2016, 2017, and 2018. Guide for maintained secondary schools, academies and free schools
Progre 8 meaure in 2016, 2017, and 2018 Guide for maintained econdary chool, academie and free chool July 2016 Content Table of figure 4 Summary 5 A ummary of Attainment 8 and Progre 8 5 Expiry or review
More informationProceedings of Power Tech 2007, July 1-5, Lausanne
Second Order Stochatic Dominance Portfolio Optimization for an Electric Energy Company M.-P. Cheong, Student Member, IEEE, G. B. Sheble, Fellow, IEEE, D. Berleant, Senior Member, IEEE and C.-C. Teoh, Student
More informationTurbulent Mixing and Chemical Reaction in Stirred Tanks
Turbulent Mixing and Chemical Reaction in Stirred Tank André Bakker Julian B. Faano Blend time and chemical product ditribution in turbulent agitated veel can be predicted with the aid of Computational
More informationCluster-Aware Cache for Network Attached Storage *
Cluter-Aware Cache for Network Attached Storage * Bin Cai, Changheng Xie, and Qiang Cao National Storage Sytem Laboratory, Department of Computer Science, Huazhong Univerity of Science and Technology,
More information6. Friction, Experiment and Theory
6. Friction, Experiment and Theory The lab thi wee invetigate the rictional orce and the phyical interpretation o the coeicient o riction. We will mae ue o the concept o the orce o gravity, the normal
More informationBi-Objective Optimization for the Clinical Trial Supply Chain Management
Ian David Lockhart Bogle and Michael Fairweather (Editor), Proceeding of the 22nd European Sympoium on Computer Aided Proce Engineering, 17-20 June 2012, London. 2012 Elevier B.V. All right reerved. Bi-Objective
More informationAcceleration-Displacement Crash Pulse Optimisation A New Methodology to Optimise Vehicle Response for Multiple Impact Speeds
Acceleration-Diplacement Crah Pule Optimiation A New Methodology to Optimie Vehicle Repone for Multiple Impact Speed D. Gildfind 1 and D. Ree 2 1 RMIT Univerity, Department of Aeropace Engineering 2 Holden
More informationCHARACTERISTICS OF WAITING LINE MODELS THE INDICATORS OF THE CUSTOMER FLOW MANAGEMENT SYSTEMS EFFICIENCY
Annale Univeritati Apuleni Serie Oeconomica, 2(2), 200 CHARACTERISTICS OF WAITING LINE MODELS THE INDICATORS OF THE CUSTOMER FLOW MANAGEMENT SYSTEMS EFFICIENCY Sidonia Otilia Cernea Mihaela Jaradat 2 Mohammad
More informationIndependent Samples T- test
Independent Sample T- tet With previou tet, we were intereted in comparing a ingle ample with a population With mot reearch, you do not have knowledge about the population -- you don t know the population
More informationProfitability of Loyalty Programs in the Presence of Uncertainty in Customers Valuations
Proceeding of the 0 Indutrial Engineering Reearch Conference T. Doolen and E. Van Aken, ed. Profitability of Loyalty Program in the Preence of Uncertainty in Cutomer Valuation Amir Gandomi and Saeed Zolfaghari
More informationRisk Management for a Global Supply Chain Planning under Uncertainty: Models and Algorithms
Rik Management for a Global Supply Chain Planning under Uncertainty: Model and Algorithm Fengqi You 1, John M. Waick 2, Ignacio E. Gromann 1* 1 Dept. of Chemical Engineering, Carnegie Mellon Univerity,
More informationMaximizing Acceptance Probability for Active Friending in Online Social Networks
Maximizing for Active Friending in Online Social Network De-Nian Yang, Hui-Ju Hung, Wang-Chien Lee, Wei Chen Academia Sinica, Taipei, Taiwan The Pennylvania State Univerity, State College, Pennylvania,
More informationSupport Vector Machine Based Electricity Price Forecasting For Electricity Markets utilising Projected Assessment of System Adequacy Data.
The Sixth International Power Engineering Conference (IPEC23, 27-29 November 23, Singapore Support Vector Machine Baed Electricity Price Forecating For Electricity Maret utiliing Projected Aement of Sytem
More informationReport 4668-1b 30.10.2010. Measurement report. Sylomer - field test
Report 4668-1b Meaurement report Sylomer - field tet Report 4668-1b 2(16) Contet 1 Introduction... 3 1.1 Cutomer... 3 1.2 The ite and purpoe of the meaurement... 3 2 Meaurement... 6 2.1 Attenuation of
More informationCASE STUDY BRIDGE. www.future-processing.com
CASE STUDY BRIDGE TABLE OF CONTENTS #1 ABOUT THE CLIENT 3 #2 ABOUT THE PROJECT 4 #3 OUR ROLE 5 #4 RESULT OF OUR COLLABORATION 6-7 #5 THE BUSINESS PROBLEM THAT WE SOLVED 8 #6 CHALLENGES 9 #7 VISUAL IDENTIFICATION
More informationBidding for Representative Allocations for Display Advertising
Bidding for Repreentative Allocation for Diplay Advertiing Arpita Ghoh, Preton McAfee, Kihore Papineni, and Sergei Vailvitkii Yahoo! Reearch. {arpita, mcafee, kpapi, ergei}@yahoo-inc.com Abtract. Diplay
More informationQueueing Models for Multiclass Call Centers with Real-Time Anticipated Delays
Queueing Model for Multicla Call Center with Real-Time Anticipated Delay Oualid Jouini Yve Dallery Zeynep Akşin Ecole Centrale Pari Koç Univerity Laboratoire Génie Indutriel College of Adminitrative Science
More informationComputing Location from Ambient FM Radio Signals
Computing Location from Ambient FM Radio Signal Adel Youef Department of Computer Science Univerity of Maryland A.V. William Building College Park, MD 20742 adel@c.umd.edu John Krumm, Ed Miller, Gerry
More informationNETWORK TRAFFIC ENGINEERING WITH VARIED LEVELS OF PROTECTION IN THE NEXT GENERATION INTERNET
Chapter 1 NETWORK TRAFFIC ENGINEERING WITH VARIED LEVELS OF PROTECTION IN THE NEXT GENERATION INTERNET S. Srivatava Univerity of Miouri Kana City, USA hekhar@conrel.ice.umkc.edu S. R. Thirumalaetty now
More informationTowards Control-Relevant Forecasting in Supply Chain Management
25 American Control Conference June 8-1, 25. Portland, OR, USA WeA7.1 Toward Control-Relevant Forecating in Supply Chain Management Jay D. Schwartz, Daniel E. Rivera 1, and Karl G. Kempf Control Sytem
More informationCASE STUDY ALLOCATE SOFTWARE
CASE STUDY ALLOCATE SOFTWARE allocate caetud y TABLE OF CONTENTS #1 ABOUT THE CLIENT #2 OUR ROLE #3 EFFECTS OF OUR COOPERATION #4 BUSINESS PROBLEM THAT WE SOLVED #5 CHALLENGES #6 WORKING IN SCRUM #7 WHAT
More informationMSc Financial Economics: International Finance. Bubbles in the Foreign Exchange Market. Anne Sibert. Revised Spring 2013. Contents
MSc Financial Economic: International Finance Bubble in the Foreign Exchange Market Anne Sibert Revied Spring 203 Content Introduction................................................. 2 The Mone Market.............................................
More informationRO-BURST: A Robust Virtualization Cost Model for Workload Consolidation over Clouds
!111! 111!ttthhh IIIEEEEEEEEE///AAACCCMMM IIInnnttteeerrrnnnaaatttiiiooonnnaaalll SSSyyymmmpppoooiiiuuummm ooonnn CCCllluuuttteeerrr,,, CCClllooouuuddd aaannnddd GGGrrriiiddd CCCooommmpppuuutttiiinnnggg
More informationOhm s Law. Ohmic relationship V=IR. Electric Power. Non Ohmic devises. Schematic representation. Electric Power
Ohm Law Ohmic relationhip V=IR Ohm law tate that current through the conductor i directly proportional to the voltage acro it if temperature and other phyical condition do not change. In many material,
More informationA New Optimum Jitter Protection for Conversational VoIP
Proc. Int. Conf. Wirele Commun., Signal Proceing (Nanjing, China), 5 pp., Nov. 2009 A New Optimum Jitter Protection for Converational VoIP Qipeng Gong, Peter Kabal Electrical & Computer Engineering, McGill
More informationSector Concentration in Loan Portfolios and Economic Capital. Abstract
Sector Concentration in Loan Portfolio and Economic Capital Klau Düllmann and Nancy Machelein 2 Thi verion: September 2006 Abtract The purpoe of thi paper i to meaure the potential impact of buine-ector
More informationAvailability of WDM Multi Ring Networks
Paper Availability of WDM Multi Ring Network Ivan Rado and Katarina Rado H d.o.o. Motar, Motar, Bonia and Herzegovina Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, Univerity
More informationA Review On Software Testing In SDlC And Testing Tools
www.ijec.in International Journal Of Engineering And Computer Science ISSN:2319-7242 Volume - 3 Iue -9 September, 2014 Page No. 8188-8197 A Review On Software Teting In SDlC And Teting Tool T.Amruthavalli*,
More informationCombining Statistics and Semantics via Ensemble Model for Document Clustering
ombining tatitic and emantic via Enemble Model or Document lutering amah Jamal Fodeh Michigan tate Univerity Eat Laning, MI, 48824 odeham@mu.edu William F Punch Michigan tate Univerity Eat Laning, MI,
More informationG*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences
Behavior Reearch Method 007, 39 (), 75-9 G*Power 3: A flexible tatitical power analyi program for the ocial, behavioral, and biomedical cience FRAZ FAUL Chritian-Albrecht-Univerität Kiel, Kiel, Germany
More informationModule 8. Three-phase Induction Motor. Version 2 EE IIT, Kharagpur
Module 8 Three-phae Induction Motor Verion EE IIT, Kharagpur Leon 33 Different Type of Starter for Induction Motor (IM Verion EE IIT, Kharagpur Inructional Objective Need of uing arter for Induction motor
More informationEvaluating Teaching in Higher Education. September 2008. Bruce A. Weinberg The Ohio State University *, IZA, and NBER weinberg.27@osu.
Evaluating Teaching in Higher Education September 2008 Bruce A. Weinberg The Ohio State Univerity *, IZA, and NBER weinberg.27@ou.edu Belton M. Fleiher The Ohio State Univerity * and IZA fleiher.1@ou.edu
More informationUtility-Based Flow Control for Sequential Imagery over Wireless Networks
Utility-Baed Flow Control for Sequential Imagery over Wirele Networ Tomer Kihoni, Sara Callaway, and Mar Byer Abtract Wirele enor networ provide a unique et of characteritic that mae them uitable for building
More informationQUANTIFYING THE BULLWHIP EFFECT IN THE SUPPLY CHAIN OF SMALL-SIZED COMPANIES
Sixth LACCEI International Latin American and Caribbean Conference for Engineering and Technology (LACCEI 2008) Partnering to Succe: Engineering, Education, Reearch and Development June 4 June 6 2008,
More informationINTERACTIVE TOOL FOR ANALYSIS OF TIME-DELAY SYSTEMS WITH DEAD-TIME COMPENSATORS
INTERACTIVE TOOL FOR ANALYSIS OF TIMEDELAY SYSTEMS WITH DEADTIME COMPENSATORS Joé Lui Guzmán, Pedro García, Tore Hägglund, Sebatián Dormido, Pedro Alberto, Manuel Berenguel Dep. de Lenguaje y Computación,
More informationLaureate Network Products & Services Copyright 2013 Laureate Education, Inc.
Laureate Network Product & Service Copyright 2013 Laureate Education, Inc. KEY Coure Name Laureate Faculty Development...3 Laureate Englih Program...9 Language Laureate Signature Product...12 Length Laureate
More informationControl of Wireless Networks with Flow Level Dynamics under Constant Time Scheduling
Control of Wirele Network with Flow Level Dynamic under Contant Time Scheduling Long Le and Ravi R. Mazumdar Department of Electrical and Computer Engineering Univerity of Waterloo,Waterloo, ON, Canada
More informationDUE to the small size and low cost of a sensor node, a
1992 IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. 14, NO. 10, OCTOBER 2015 A Networ Coding Baed Energy Efficient Data Bacup in Survivability-Heterogeneou Senor Networ Jie Tian, Tan Yan, and Guiling Wang
More informationExploiting A Support-based Upper Bound of Pearson s Correlation Coefficient for Efficiently Identifying Strongly Correlated Pairs
Exploiting A Support-baed Upper Bound of Pearon Correlation Coefficient for Efficiently Identifying Strongly Correlated Pair Hui Xiong Computer Science Univerity of Minneota huix@c.umn.edu Shahi Shekhar
More informationPerformance of a Browser-Based JavaScript Bandwidth Test
Performance of a Brower-Baed JavaScript Bandwidth Tet David A. Cohen II May 7, 2013 CP SC 491/H495 Abtract An exiting brower-baed bandwidth tet written in JavaScript wa modified for the purpoe of further
More informationAssigning Tasks for Efficiency in Hadoop
Aigning Tak for Efficiency in Hadoop [Extended Abtract] Michael J. Ficher Computer Science Yale Univerity P.O. Box 208285 New Haven, CT, USA michael.ficher@yale.edu Xueyuan Su Computer Science Yale Univerity
More informationGroup Mutual Exclusion Based on Priorities
Group Mutual Excluion Baed on Prioritie Karina M. Cenci Laboratorio de Invetigación en Sitema Ditribuido Univeridad Nacional del Sur Bahía Blanca, Argentina kmc@c.un.edu.ar and Jorge R. Ardenghi Laboratorio
More informationAN OVERVIEW ON CLUSTERING METHODS
IOSR Journal Engineering AN OVERVIEW ON CLUSTERING METHODS T. Soni Madhulatha Aociate Preor, Alluri Intitute Management Science, Warangal. ABSTRACT Clutering i a common technique for tatitical data analyi,
More informationUnobserved Heterogeneity and Risk in Wage Variance: Does Schooling Provide Earnings Insurance?
TI 011-045/3 Tinbergen Intitute Dicuion Paper Unoberved Heterogeneity and Rik in Wage Variance: Doe Schooling Provide Earning Inurance? Jacopo Mazza Han van Ophem Joop Hartog * Univerity of Amterdam; *
More informationHow Enterprises Can Build Integrated Digital Marketing Experiences Using Drupal
How Enterprie Can Build Integrated Digital Marketing Experience Uing Drupal acquia.com 888.922.7842 1.781.238.8600 25 Corporate Drive, Burlington, MA 01803 How Enterprie Can Build Integrated Digital Marketing
More informationPOSSIBILITIES OF INDIVIDUAL CLAIM RESERVE RISK MODELING
POSSIBILITIES OF INDIVIDUAL CLAIM RESERVE RISK MODELING Pavel Zimmermann * 1. Introduction A ignificant increae in demand for inurance and financial rik quantification ha occurred recently due to the fact
More informationMBA 570x Homework 1 Due 9/24/2014 Solution
MA 570x Homework 1 Due 9/24/2014 olution Individual work: 1. Quetion related to Chapter 11, T Why do you think i a fund of fund market for hedge fund, but not for mutual fund? Anwer: Invetor can inexpenively
More informationProject Management Basics
Project Management Baic A Guide to undertanding the baic component of effective project management and the key to ucce 1 Content 1.0 Who hould read thi Guide... 3 1.1 Overview... 3 1.2 Project Management
More informationGrowing Self-Organizing Maps for Surface Reconstruction from Unstructured Point Clouds
Growing Self-Organizing Map for Surface Recontruction from Untructured Point Cloud Renata L. M. E. do Rêgo, Aluizio F. R. Araújo, and Fernando B.de Lima Neto Abtract Thi work introduce a new method for
More informationReturn on Investment and Effort Expenditure in the Software Development Environment
International Journal of Applied Information ytem (IJAI) IN : 2249-0868 Return on Invetment and Effort Expenditure in the oftware Development Environment Dineh Kumar aini Faculty of Computing and IT, ohar
More informationBrand Equity Net Promoter Scores Versus Mean Scores. Which Presents a Clearer Picture For Action? A Non-Elite Branded University Example.
Brand Equity Net Promoter Score Veru Mean Score. Which Preent a Clearer Picture For Action? A Non-Elite Branded Univerity Example Ann Miti, Swinburne Univerity of Technology Patrick Foley, Victoria Univerity
More informationImproving the Performance of Web Service Recommenders Using Semantic Similarity
Improving the Performance of Web Service Recommender Uing Semantic Similarity Juan Manuel Adán-Coello, Carlo Miguel Tobar, Yang Yuming Faculdade de Engenharia de Computação, Pontifícia Univeridade Católica
More informationMobility Improves Coverage of Sensor Networks
Mobility Improve Coverage of Senor Networ Benyuan Liu Dept. of Computer Science Univerity of Maachuett-Lowell Lowell, MA 1854 Peter Bra Dept. of Computer Science City College of New Yor New Yor, NY 131
More informationMobile Network Configuration for Large-scale Multimedia Delivery on a Single WLAN
Mobile Network Configuration for Large-cale Multimedia Delivery on a Single WLAN Huigwang Je, Dongwoo Kwon, Hyeonwoo Kim, and Hongtaek Ju Dept. of Computer Engineering Keimyung Univerity Daegu, Republic
More informationSCM- integration: organiational, managerial and technological iue M. Caridi 1 and A. Sianei 2 Dipartimento di Economia e Produzione, Politecnico di Milano, Italy E-mail: maria.caridi@polimi.it Itituto
More informationSELF-MANAGING PERFORMANCE IN APPLICATION SERVERS MODELLING AND DATA ARCHITECTURE
SELF-MANAGING PERFORMANCE IN APPLICATION SERVERS MODELLING AND DATA ARCHITECTURE RAVI KUMAR G 1, C.MUTHUSAMY 2 & A.VINAYA BABU 3 1 HP Bangalore, Reearch Scholar JNTUH, Hyderabad, India, 2 Yahoo, Bangalore,
More informationRisk-Sharing within Families: Evidence from the Health and Retirement Study
Rik-Sharing within Familie: Evidence from the Health and Retirement Study Ş. Nuray Akın and Okana Leukhina December 14, 2014 We report trong empirical upport for the preence of elf-interet-baed rik haring
More informationAuction-Based Resource Allocation for Sharing Cloudlets in Mobile Cloud Computing
1 Auction-Baed Reource Allocation for Sharing Cloudlet in Mobile Cloud Computing A-Long Jin, Wei Song, Senior Member, IEEE, and Weihua Zhuang, Fellow, IEEE Abtract Driven by pervaive mobile device and
More informationTwo Dimensional FEM Simulation of Ultrasonic Wave Propagation in Isotropic Solid Media using COMSOL
Excerpt from the Proceeding of the COMSO Conference 0 India Two Dimenional FEM Simulation of Ultraonic Wave Propagation in Iotropic Solid Media uing COMSO Bikah Ghoe *, Krihnan Balaubramaniam *, C V Krihnamurthy
More informationBundled Discounts: Strategic Substitutes or Complements?
Bundled Dicount: Strategic Subtitute or Complement? Duarte Brito y Univeridade Nova de Liboa and CEFGE-UE Helder Vaconcelo z Faculdade de Economia, Univeridade do Porto, CEF.UP and CEPR June 2, 24 btract
More informationSoftware Engineering Management: strategic choices in a new decade
Software Engineering : trategic choice in a new decade Barbara Farbey & Anthony Finkeltein Univerity College London, Department of Computer Science, Gower St. London WC1E 6BT, UK {b.farbey a.finkeltein}@ucl.ac.uk
More informationTrusted Document Signing based on use of biometric (Face) keys
Truted Document Signing baed on ue of biometric (Face) Ahmed B. Elmadani Department of Computer Science Faculty of Science Sebha Univerity Sebha Libya www.ebhau.edu.ly elmadan@yahoo.com ABSTRACT An online
More informationOnline story scheduling in web advertising
Online tory cheduling in web advertiing Anirban Dagupta Arpita Ghoh Hamid Nazerzadeh Prabhakar Raghavan Abtract We tudy an online job cheduling problem motivated by toryboarding in web advertiing, where
More informationMATLAB/Simulink Based Modelling of Solar Photovoltaic Cell
MATLAB/Simulink Baed Modelling of Solar Photovoltaic Cell Tarak Salmi *, Mounir Bouzguenda **, Adel Gatli **, Ahmed Mamoudi * *Reearch Unit on Renewable Energie and Electric Vehicle, National Engineering
More informationHUMAN CAPITAL AND THE FUTURE OF TRANSITION ECONOMIES * Michael Spagat Royal Holloway, University of London, CEPR and Davidson Institute.
HUMAN CAPITAL AND THE FUTURE OF TRANSITION ECONOMIES * By Michael Spagat Royal Holloway, Univerity of London, CEPR and Davidon Intitute Abtract Tranition economie have an initial condition of high human
More informationThe Arms Race on American Roads: The Effect of SUV s and Pickup Trucks on Traffic Safety
The Arm Race on American Road: The Effect of SUV and Pickup Truck on Traffic Safety Michelle J. White Univerity of California, San Diego, and NBER Abtract Driver have been running an arm race on American
More informationBio-Plex Analysis Software
Multiplex Supenion Array Bio-Plex Analyi Software The Leader in Multiplex Immunoaay Analyi Bio-Plex Analyi Software If making ene of your multiplex data i your challenge, then Bio-Plex data analyi oftware
More informationMECH 2110 - Statics & Dynamics
Chapter D Problem 3 Solution 1/7/8 1:8 PM MECH 11 - Static & Dynamic Chapter D Problem 3 Solution Page 7, Engineering Mechanic - Dynamic, 4th Edition, Meriam and Kraige Given: Particle moving along a traight
More informationCombining Supervised and Unsupervised Monitoring for Fault Detection in Distributed Computing Systems
Combining Supervied and Unupervied Monitoring for Fault Detection in Ditributed Computing Sytem Haifeng Chen Guofei Jiang Critian Ungureanu Kenji Yohihira NEC Laboratorie America, Inc. Independence Way
More informationRISK MANAGEMENT POLICY
RISK MANAGEMENT POLICY The practice of foreign exchange (FX) rik management i an area thrut into the potlight due to the market volatility that ha prevailed for ome time. A a conequence, many corporation
More information