# So Who Won? Dynamic Max Discovery with the Crowd

Save this PDF as:

Size: px
Start display at page:

## Transcription

3 Since we know the values of both p and W, we can derive a formula for P(π d W) in Equation. In particular, the most likely permutation(s), is simply: W = 3 C A Figure : How should these objects be ranked? Vote matrix (left) and equivalent graph representation (right). Arc weights indicate number of votes. are made about the structure of matrix W. The evidence can also be viewed as a directed weighted graph G v = (V, A), with the vertices being the objects and the arcs representing the vote outcomes. For each pair (i, j), if w ij >, arc (i, j) with weight w ij is present in A. For example, Figure displays a sample vote matrix and equivalent graph representation. In this example, object is called A, object is B, and so on. For instance, there is an arc from vertex (object) B to vertex C with weight because w,3 =, and there is a reverse arc from C to B with weight because w 3, =. If there are no votes (w ij =, as from B to A), we can either say that there is no arc, or that the arc has weight.. Maximum Likelihood Preliminaries: We first present a Maximum Likelihood (ML) formulation of the Judgment Problem. We directly compute the object that has the highest probability of being the maximum object in O, given vote matrix W. Assuming that average worker accuracy p is known, the ML formulation we present is the optimal feasible solution to the Judgment Problem. Let π be a random variable over the set of all n! possible permutations, where we assume a-priori that each permutation is equally likely to be observed. We denote the probability of a given permutation π d given the vote matrix W as P(π = π d W). For the ease of exposition, we adopt the shorthand P(π d W) instead of writing P(π = π d W). To derive the formula for P(π d W), we first apply Bayes theorem, P(π d W) = P(W π d)p(π d ) P(W) = P(W π d)p(π d ) P(W π j)p(π j) From our assumption that the prior probabilities of all permutations are equal, P(π d ) = n!. Now consider P(W π d ). Given a permutation π d, for each unordered pair {i, j}, the probability f πd (i, j) of observing w ij and w ji is the binomial distribution probability mass function (p.m.f.): ( `wij +w ji pw ji w f πd (i, j) = ij ( p) w ij if π d (i) < π d (j) `wij +w ji pw ij w ji ( p) w ji if π d (j) < π d (i) () Note that if both w ij and w ji are equal to, then f πd (i, j) =. Now, given a permutation π d, observing the votes involving an unordered pair {i, j} is conditionally independent of observing the votes involving any other unordered pair. Using this fact, P(W π d ), the probability of observing all votes given a permutation π d is simply: P(W π d ) = Y f πd (i, j) (3) i,j:i<j j () arg max P(π d W) (4) d The permutations optimizing Equation 4 are also known as Kemeny permutations or Kemeny rankings [9]. For example, consider the matrix W of Figure. We do not show the computations here, but it turns out that the two most probable permutations of the objects are (D, C, B, A) and (C, D, B,A), with all other permutations having lower probability. This result roughly matches our intuition, since object A was never voted to be greater than any of the other objects, and C and D have more votes in favor over B. We can derive the formula for the probability that a given object o j has a given rank k. Let π (i) denote the position of object i in the permutation associated with random variable π. We are interested in the probability P(π (k) = j W). Since the event (π = π d ) is disjoint for different permutations π d, we have: P(π (k) = j W) = P(π d W) d:π d (k)=j Substituting for P(π d W) using Equation and simplifying, we have: P(W π d ) P(π d:π d (k) = j W) = (k)=j (5) P(W π l ) Since we are interested in the object o j with the highest probability of being rank, e.g., P(π () = j W), we now have the Maximum Likelihood formulation to the Judgment Problem: ML FORMULATION (JUDGMENT). Given W and p, determine: arg max j P(π () = j W). In the example graph of Figure, while C and D both have Kemeny permutations where they are the greatest objects, D is the more likely max over a large range of p values. For instance, for p =.75, P(π () = C W) =.36 while P(π () = D W) =.54. This also matches our intuition, since C has one vote where it is less than B, while D is never voted to be less than either A or B. Maximum Likelihood Strategy: Equation 5 implies that we only need to compute P(W π d ) for each possible permutation π d, using Equation 3, in order to determine P(π (k) = j W) for all values j and k. In other words, by doing a single pass through all permutations, we can compute the probability that any object o j has a rank k, given the vote matrix W. We call this exhaustive computation of probabilities the Maximum Likelihood Strategy and use it as a baseline in our experiments. Note that the ML strategy is the optimal feasible solution to the Judgment Problem. The pseudocode for this strategy is displayed in Strategy. The strategy utilizes a ML scoring function, which computes the score of each object as s(j) = P(π () = j W). The predicted max is then the object with the highest score. Note that the strategy can be easily adapted to compute the maximum likelihood of arbitrary ranks (not just first) over the set of all possible permutations..3 Computational Complexity l

4 Maximum Likelihood Permutations: We begin by considering the complexity of computing Equation 4. The problem has been shown to be NP-Hard [3]. We briefly describe the result, before moving on to the Judgment Problem. Consider Equation 3. Let ψ(π d, W) denote the number of votes in W that agree with permutation π d, e.g. ψ(π d, W) = ij:π d (j)<π d (i) Let T(W) denote the total number of votes in W, e.g. T = W. Equation 3 can be rewritten as: P(W π d ) = Zp ψ(π d,w) ( p) T(W) ψ(π d,w) (6) w ij. In this expression, Z is a constant. Note that p x ( p) a x is an increasing function with respect to x, for.5 < p < and constant a. Therefore, maximizing P(W π d ) is equivalent to maximizing ψ(π d, W). This implies that computing arg max i ψ(π i, W) is equivalent to computing the most likely permutation(s), arg max i P(π i W). ψ( ) is known as the Kemeny metric, and has been well studied in the economic and social choice literature [9]. Referring to the example in Figure, the Kemeny metric is maximized by two permutations of the objects, (D, C, B, A) or (C, D, B, A). For both these permutations, the Kemeny metric is equal to +++3 = 8. We next show that maximizing the Kemeny metric is equivalent to solving a classical NP-Hard problem. Consider the directed graph G v = (V, A) representing the votes of W (Figure is an example). The minimum feedback arc set of G v is the smallest weight set of arcs A A such that (V, A\A ) is acyclic. Equivalently, the problem can also be seen as maximizing the weight of acyclic graph (V, A\A ). Now, supposing we have a method to solve for the minimum feedback arc set, consider a topological ordering π t of the vertices in the resulting acyclic graph. Referring back to Equation 6, the permutation π t maximizes the Kemeny metric p ψ(π,w). Therefore, solving the minimum feedback arc set problem for G v is equivalent to our original problem of finding the most likely permutation given a set of votes. Referring back to our example in Figure, the minimum feedback arc set is, equivalent to cutting arc (C, B) and one of arcs (C, D) or (D, C). Finding the minimum feedback arc set in a directed graph is a classical NP-Hard problem, implying that the problem of finding the ML permutation given a set of votes is NP-Hard as well. THEOREM. (Hardness of Maximum Likelihood Permutation) [3] Finding the Maximum Likelihood permutation given evidence is NP-Hard. Hardness of the Judgment Problem: In Section., we presented a formulation for the Judgment Problem based on ML for finding the object most likely to be the max (maximum) object in O. Unfortunately, the strategy based on that formulation was computationally infeasible, as it required computation across all n! permutations of the objects in O. We now show that the optimal solution to the problem of finding the maximum object is in fact NP-Hard using a reduction from the problem of determining Kemeny winners [7]. (Hudry et al. [7] actually show that determining Slater winners in tournaments is NP-Hard, but their proof also holds for Kemeny winners. We will describe the Kemeny winner problem below.) Our results and proof are novel. THEOREM. (Hardness of the Judgment Problem) Finding the maximum object given evidence is NP-Hard. OOF. We first describe the Kemeny winner problem. In this proof, we use an alternate (but equivalent) view of a directed weighted graph like Figure. In particular, we view weighted arcs as multiple arcs. For instance, if there is an arc from vertex A to B with weight 3, we can instead view it as 3 separate edges from A to B. We use this alternate representation in our proof. An arc i j respects a permutation if the permutation has o j ranked higher than o i (and does not if the permutation has o i ranked higher than o j). A Kemeny permutation is simply a permutation of the vertices (objects), such that the number of arcs that do not respect the permutation is minimum. There may be many such permutations, but there always is at least one such permutation. The starting vertex (rank object) in any of these permutations is a Kemeny winner. It can be shown that finding a Kemeny winner is NP-Hard (using a reduction from the feedback arc set problem, similar to the proof in Hudry et al. [7]). We now reduce the Kemeny winner determination problem to one of finding the maximum object. Consider a directed weighted graph G, where we wish to find a Kemeny winner. We show that with a suitable probability p, which we set, the maximum object (i.e., the solution to the Judgment Problem) in G is a Kemeny winner. As before, the probability that a certain object o j is the maximum object is the right hand side of Equation 5 with k set to. The denominator can be ignored since it is a constant for all j. We set worker accuracy p to be very close to. In particular, we choose a value p such that p <. p n! Now, consider all permutations π d that are not Kemeny permutations. In this case, it can be shown that d:π d is not Kemeny P(W π d ) < d:π d is not Kemeny P(W π s) for any Kemeny permutation π s. Thus, the object o j that maximizes Equation 5 (for k = ) has to be one that is a Kemeny winner. To see why P(W π d ) < P(W π s) for a Kemeny permutation π s, notice that the left hand side is at most n! P(W π d) where π d is the permutation (not Kemeny) that has the least number of arcs that do not respect the permutation. Note that P(W π d) is at most P(W π s) p, since this permutation has at p least one more mistake as compared to any Kemeny permutation. Therefore, we have shown that, for a suitable p, the maximum object in G is a Kemeny winner. Thus, we have a reduction from the Kemeny winner problem to the Judgement problem. Since finding a Kemeny winner is NP-Hard, this implies that finding the maximum object in G is NP-Hard. #P-Hardness of Probability Computations: In addition to being NP-Hard to find the maximum object, we can show that evaluating the numerator of the right hand side of Equation 5 (with k = ) is #P-Hard, in other words: computing P(π () = j, W) is #P- Hard. We use a reduction from the problem of counting the number of linear extensions in a directed acyclic graph (DAG), which is known to be #P-Hard. THEOREM 3. (#P-Hardness of Probability Computation) Computing P(π () = j, W) is #P-Hard. OOF. A linear extension is a permutation of the vertices, such that all arcs in the graph respect the permutation (i.e., a linear extension is the same as a Kemeny permutation for a DAG). Consider a DAG G = (V, A). We add an additional vertex x such that there is an arc from each of the vertices in G to x, giving a new graph G = (V, A ). We now show that computing P(π () = x,w) in G can be used to compute the number of

5 linear extensions in G. Notice that: A P(π () = x,w) = a ip i ( p) A i i= A = p A a p i i( p ) A (7) where a i is the number of permutations where there are i arcs that respect the permutation. Clearly, the number that we wish to determine is a A, since that is the number of permutations that correspond to linear extensions. Equation 7 is a polynomial of degree A in p, thus, we may simply choose p A + different values of p, generate p A + different graphs G, and use the probability computation in Equation 7 to create a set of A + equations involving the a i coefficients. We may then derive the value of a A using Lagrange s interpolation formula. Since vertex x is the only maximum vertex in G, by computing P(π () = x, W) in G, we count the number of linear extensions in DAG G. Since counting the number of linear extensions in a DAG is #P-Hard, this implies that the computation of P(π () = x, W) in G is #P-Hard, which implies that the computation of P(π (k) = j, W) for directed graph G v (associated with vote matrix W ) is #P-Hard. Strategy Maximum Likelihood Require: n objects, probability p, vote matrix W Ensure: ans = maximum likelihood maximum object s[ ] {s[i] is used to accumulate the probability that i is the maximum object} for each permutation π of n objects do prob {prob is the probability of permutation π given vote matrix W } for each tuple (i, j) : i < j do if π(i) < π(j) then prob prob `w ij +w ji pw ji w ij ( p) w ij else i= prob prob `w ij +w ji pw ij w ji ( p) w ji end if s[π ()] s[π ()] + prob ans argmax i s[i].4 Heuristic Strategies The ML scoring function is computationally inefficient and also requires prior knowledge of p, the average worker accuracy, which is not available to us in real-world scenarios. We next investigate the performance and efficiency of four heuristic strategies, each of which runs in polynomial time. The heuristics we present, excluding the Indegree heuristic, do not require explicit knowledge of the worker accuracy p. Indegree Strategy: The first heuristic we consider is an Indegree scoring function proposed by Coppersmith et al. [] to approximate the optimal feedback arc set in a directed weighted graph where arc weights l ij, l ji satisfy l ij + l ji = for each pair of vertices i and j. In this section, we describe how to transform our vote matrix W to a graph where this Indegree scoring function can be applied. Let π(i) denote the rank, or index, of object o i in the permutation associated with random variable π. Given vote matrix W, we construct a complete graph between all objects where arc weights l ji are equal to P(π(i) < π(j) w ij, w ji). l ji reflects the probability that o i is greater than o j given the local evidence w ij and w ji. It is important to note that this method is a heuristic, e.g., we compute P(π(i) < π(j) w ij, w ji), rather than P(π(i) < π(j) W), which requires full enumeration over all n! permutations. How do we compute arc weight P(π(i) < π(j) w ij, w ji)? P(π(i) < π(j) w ij, w ji) = P(w ij, w ji π(i) < π(j))p(π(i) < π(j)) P(w ij, w ji) (8) Assuming that a priori all permutations π are equally likely, P(π(i) < π(j)) = P(π(j) < π(i)) by symmetry. Using Equation 8 to find expressions for P(π(i) < π(j) w ij, w ji) and P(π(i) > π(j) w ij, w ji), we can derive the following: P(π(i) < π(j) w ij, w ji) P(wij, wji π(i) < π(j)) = P(π(i) > π(j) w ij, w ji) P(w ij, w ji π(i) > π(j)) Since P(π(i) < π(j) w ij, w ji)+p(π(j) < π(i) w ij, w ji) =, we can simplify Equation 9 to get the following expression: P(π(i) < π(j) w ij, w ji) = P(w ij, w ji π(i) < π(j)) P(w ij, w ji π(i) < π(j)) + P(w ij, w ji π(i) > π(j)) (9) () Using Equation, P(w ij, w ji π(i) < π(j)) and P(w ij, w ji π(i) < π(j)) can be computed directly from the binomial distribution p.m.f. Therefore, we can compute the arc weight l ji = P(π(i) < π(j) w ij, w ji) needed for the Indegree scoring function. It should be clear that l ij +l ji =. Also, if w ij and w ji are both equal to zero, then both l ij and l ji are computed to be.5. Strategy Indegree Require: n objects, probability p, vote matrix W Ensure: ans = predicted maximum object s[ ] for i :... n do for j :... n, j i do s[i] s[i] + l ji {l ji = P(π(i) < π(j) w ij, w ji)} ans argmax i s[i] The Indegree scoring function, displayed in Strategy, computes the score of object o j as: s(j) = P i lij. Intuitively, vertices with higher scores correspond to objects which have compared favorably to other objects, and hence should be ranked higher. The predicted ranking has been shown to be a constant factor approximation to the feedback arc set for directed graphs where all arcs (i, j) are present and l ij + l ji = []. The running time of this heuristic is dominated by the time to do the final sort of the scores. Let us walk through the example graph in Figure. First, for those pairs of vertices that do not have any votes between them, we have l AC =.5, l CA =.5, l AD =.5, and l DA =.5. By symmetry, l CD =.5 and l DC =.5. Given a value of p, we use Equation to compute the rest of the arc weights. For p =.55, we have l AB =.599, l BA =.4, l BC =.55, l CB =.45, l BD =.646, and l DB =.354. With these computed arc weights, we obtain the scores: s(a) =.4, s(b) =.43, s(c) =.55, and s(d) =.65, generating a predicted ranking of (D, C, B, A),

6 with object D being the predicted maximum object. Note that if p is larger, e.g. p =.95, the Indegree heuristic predicts the same ranking. Local Strategy: The Indegree heuristic is simple to compute, but only takes into account local evidence. That is, the score of object o i only depends on the votes that include o i directly. We now consider a Local scoring function, adapted from a heuristic proposed by David [3], which considers evidence two steps away from o i. This method was originally proposed to rank objects in incomplete tournaments with ties. We adapted the scoring function to our setting, where there can be multiple comparisons between objects, and there are no ties in comparisons. This heuristic is based on the notion of wins and losses, defined as follows: wins(i) = P j wji and losses(i) = P i wij. For instance, in Figure, vertex B has 3 wins and 5 losses. The score s(i) has three components. The first is simply wins(i) losses(i), reflecting the net number of votes in favor of o i. For vertex B, this first component would be 3 5 =. Since this first component does not reflect the strength of the objects o i was compared against, we next add a reward : for each o j such that w ji > w ij (i has net wins over j), we add wins(j) to the score of o i. In our example, B only has net wins over A, so we reward B with wins(a) (which in this case is zero). On the other hand, since C beat out B, then C gets a reward of wins(b) = 3 added to its score. Finally, we penalize s(i) by subtracting losses(j) for each o j that overall beat o i. In our example, we subtract from s(b) both losses(c) = and losses(d) =. Thus, the final score s(b) is plus the reward minus the penalty, i.e., s(b) = + 3 = 5. More formally, score s(i) is defined as follows: s(i) = wins(i) losses(i) + j j [(w ji > w ij)wins(j)] [(w ij > w ji)losses(j)] () Strategy 3 Local Require: n objects, vote matrix W Ensure: ans = predicted maximum object wins[ ], losses[ ], s[ ] {objects are ranked by s[ ]} for each tuple (i, j) : do wins[j] wins[j] + w ij losses[i] losses[i] + w ij for i :... n do s[i] wins[i] losses[i] {add wins losses to s} for j :... n, j i do if w ij < w ji then s[i] s[i] + wins[j] {add reward} else if w ij > w ji then s[i] s[i] losses[j] {subtract penalty} end if ans argmax i s[i] Having computed s( ), we sort all objects by decreasing order of s. The resulting permutation is our predicted ranking, with the vertex having largest s being our predicted maximum object. An implementation of the method is displayed in Strategy 3. To complete the example of Figure, Strategy 3 computes the following scores: s(a) = 5 = 7, s(b) = = 5, s(c) = = 4, and s(d) = 4 +3 = 6. The predicted ranking is then (D, C, B, A), with object D being the predicted maximum object. PageRank Strategy: Both the Indegree and Local heuristics use only information one or two steps away to make inferences about the objects of O. We next consider a global heuristic scoring function inspired by the PageRank [6] algorithm. The general idea behind using a PageRank-like procedure is to utilize the votes in W as a way for objects to transfer strength between each other. The use of PageRank to predict the maximum has been previously considered [5] in the literature. Our contribution is a modified PageRank to predict the maximum object in O, which in particular, can handle directed cycles in the directed graph representing W. Consider again the directed graph G v representing the votes of W (Figure is an example). Let d + (i) to denote the outdegree of vertex i in G v, e.g. d + (i) = P j wij. If d+ (i) =, we say that i is a sink vertex. Let pr t(i) represent the PageRank of vertex i in iteration t. We initialize each vertex to have the same initial PageRank, e.g., pr ( ) =. In each iteration t +, we apply the following update equation n to each vertex i: pr t+(i) = w ji prt(j) () d + (j) j For each iteration, each vertex j transfers all its PageRank (from the previous iteration) proportionally to the other vertices i whom workers have indicated may be greater than j, where the proportion of j s PageRank transferred to i is equal to w ji d + (j). Intuitively, pr t(i) can be thought as a proxy for the probability that object o i is the maximum object in O (during iteration t). What happens to the PageRank vector after performing many update iterations using Equation? Considering the strongly connected components (SCCs) of G v, let us define a terminal SCC to be a SCC whose vertices do not have arcs transitioning out of the SCC. After a sufficient number of iterations, the PageRank probability mass in G v becomes concentrated in the terminal SCCs of G v, with all other vertices outside of these SCCs having zero PageRank [4]. In the context of our problem, these terminal SCCs can be thought of as sets of objects which are ambiguous to order. Our proposed PageRank algorithm is described in Strategy 4. How is our strategy different from the standard PageRank algorithm? The original PageRank update equation is: pr t+(i) = d n + d j w ji d + (j) prt(j)) Comparing the original equation and Equation, the primary difference is that we use a damping factor d =, e.g. we remove jump probabilities. PageRank was designed to model the behavior of a random surfer traversing the web, while for the problem of ranking objects, we do not need to model a random jump vector. A second difference between our modified PageRank and the original PageRank is that prior to performing any update iterations, for each sink vertex i, we set w ii equal to in W. In our setting, sinks correspond to objects which may be the maximum object (e.g., no worker voted that o i is less than another object). By setting w ii to initially, from one iteration to the next, the PageRank in sink i remains in sink i. This allows PageRank to accumulate in sinks. Contrast this with the standard PageRank methodology, where when a random surfer reaches a sink, it is assumed that (s)he transitions to all other vertices with equal probability. Finally, a caveat to our PageRank strategy is that the PageRank vector (pr( ) in Strategy 4) may not converge for some vertices

7 Strategy 4 PageRank Require: n objects, vote matrix W, K iterations Ensure: ans = predicted maximum object construct G v = (V, A) from W compute d + [ ] for each vertex {compute all outdegrees} for i :... n do if d + [i] == then w ii end if pr [ ] n {pr is the PageRank vector in iteration } for k :... K do for i :... n do for j :... n, j i do pr k [i] pr k [i] + w ji d + [j] pr k [j] compute period[ ] of each vertex using final iterations of pr[ ] for i :... n do s[i] {s[ ] is a vector storing average PageRank} for j :... period[i] do s[i] s[i] + pr K j [i] s[i] s[i] period[i] ans argmax i s[i] in terminal SCCs. To handle the oscillating PageRank in terminal SCCs, we execute our PageRank update equation (Equation ) for a large number of iterations, denoted as K in Strategy 4. Then, we examine the final iterations, say final %, of the PageRank vector to empirically determine the period of each vertex, where we define the period as the number of iterations for the PageRank value of a vertex to return to its current value. In practice, we find that running PageRank for K iterations, where K = O(n), is sufficient to detect the period of nearly all vertices in terminal SCCs. For example, consider a graph among 3 objects A, B, C with 3 arcs: (A, B),(B, C), and (C, B). All vertices initially have PageRank probability. After iteration, the PageRank vector is (,, ) After iterations, the PageRank vector is (,, ). And so on. In 3 3 this example, object B and C each have periods of. With the periods computed for each vertex, we compute an average PageRank value for each vertex over its period. This average PageRank is used as the scoring function s( ) for this strategy. After the termination of PageRank, we sort the vertices by decreasing order of s( ), and predict that the vertex with maximum average PageRank corresponds to the maximum object in O. Note that our PageRank heuristic is primarily intended to predict a maximum object, not to predict a ranking of all objects (as many objects will end up with no PageRank). However, for completeness, when evaluating PageRank in later experiments, we still do consider the predicted ranking induced by PageRank. The details of our implementation are displayed in Strategy 4. To illustrate our PageRank heuristic, consider again the example in Figure. There are SCCs in the graph: (A) and (B, C, D), with (B, C, D) being a terminal SCC. Each of the 4 vertices is initialized with.5 PageRank. After the first iteration, the PageRank vector is (,.375,.35,.75). After the second iteration, the PageRank vector is (,.375,.35,.75). After ~ iterations, the PageRank vector oscillates around (,.7,.435,.348). With a sufficiently large number of iterations and an appropriately chosen convergence threshold, the heuristic determines a period of for both SCCs and computes an average PageRank vector of (,.7,.435,.348). The PageRank heuristic then predicts ob- Heuristic Prediction ML (D, C, B, A) and (C, D, B, A) Indegree (D, C, B, A) Local (D, C, B, A) PageRank Maximum object = C Iterative (C, D, B, A),(C, D, A,B),(D, C, B, A), or (D, C, A,B) Table : Predictions using each heuristic for Figure. ject C to be the maximum object in O. Iterative Strategy: We next propose an Iterative heuristic strategy to determine the maximum object in O. The general framework is the following:. Place all objects in a set.. Rank the objects in the set by a scoring metric. 3. Remove the lower ranked objects from the set. 4. Repeat steps 3 and 4 until only one object remains. There are two parameters we can vary in this framework: the scoring metric and the number of objects eliminated each iteration. Let us define the dif(i) metric of object o i to be equal to wins(i) losses(i). An implementation of the Iterative strategy using the dif metric is displayed in Strategy 5. In our particular implementation, we emphasize computational efficiency and remove half of the remaining objects each iteration. The Iterative strategy relies upon the elimination of lower ranked objects before reranking higher ranked objects. With each iteration, as more objects are removed, the difs of the higher ranked objects separate from the difs of the lower ranked objects. Basically, by removing lower ranked objects, the strategy is able to more accurately rank the remaining set of objects. The strategy can be thought of as iteratively narrowing in on the maximum object. It is important to note that other scoring metrics can be used with this Iterative strategy as well. For example, by iteratively ranking with the Local heuristic, we were able to achieve (slightly) better performance than the simple dif metric. Our method is similar to the Greedy Order algorithm proposed by Cohen et al. [7], who considered a problem related to feedback arc set. Our strategy differs in that it is more general (e.g., it can utilize multiple metrics), and our strategy can be optimized (e.g., if we eliminate half of the objects each iteration, we require only a logarithmic number of sorts, as opposed to a linear number). The Iterative strategy can also be viewed as a scoring function s( ), like the prior heuristics we have examined. Denoted as s[ ] in Strategy 5, we can assign each object a score equal to the iteration number in which it was removed from set T. Using this scoring function s( ), the predicted maximum object is then simply argmax i s(i). Returning to the example graph in Figure, the Iterative heuristic first computes the dif metric for each object: dif(a) =, dif(b) =, dif(c) = and dif(d) = 3. The objects are then placed in a set and sorted by dif. In the first iteration, objects A and B are assigned ranks 3 and 4 and removed from the set. Then, dif is recomputed among all remaining objects in the set, dif(c) = and dif(d) =. In the second iteration, either object C or D is removed and assigned rank. In the third iteration, the remaining object is removed and assigned rank. Therefore, the predicted ranking of the Iterative heuristic is equally likely to be (C, D, B, A),(C, D, A,B),(D, C, B,A), or (D, C, A, B), with the predicted maximum object of the heuristic being object C or D with equal probability. To summarize, Table displays the predictions for ML and our four heuristics for the example displayed in Figure. How can we determine which heuristic strategy is superior to the others?

8 P at Edge Coverage P at Edge Coverage P at Edge Coverage Figure 3: Precision at versus Edge Coverage. objects. p=.55 (left), p=.75 (middle), p=.95 (right). Strategy 5 Iterative Require: n objects, vote matrix W Ensure: ans = predicted maximum object dif[ ] {dif[ ] is the scoring metric} for i :... n do for j :... n, j i do dif[j] dif[j] + w ij ; dif[i] dif[i] w ij initialize set Q {which stores objects} for i :... n do Q Q i while Q > do sort objects in Q by dif[ ] for r : ( Q + )... Q do remove object i (with rank r) from Q for j : j Q do if w ij > then dif[j] dif[j] w ij ; dif[i] dif[i] + w ij end if if w ji > then dif[i] dif[i] w ji ; dif[j] dif[j] + w ji end if end while ans S[] {S[] is the final object in S} P at ML Edge Coverage MRR ML Edge Coverage Figure : Comparison of ML and heuristics. Prediction performance versus Edge Coverage. 5 objects, p=.75. (left), MRR (right)..5 Experiments In this section, we experimentally compare our heuristic strategies: Indegree (), Local (), PageRank, (), and Iterative (). We also compare them with the Maximum Likelihood (ML) Strategy, which we consider the best possible way to select the maximum. However, since ML is computationally very expensive, we only do this comparison on a small scenario. For our experiments, we synthetically generate problem instances, varying : n (the number of objects in O), v (the number of votes we sample for W ), and p (average worker accuracy). We prefer to use synthetic data, since it lets us study a wide spectrum of scenarios, with highly reliable or unreliable workers, and with many or few votes. In our base experiments, we vary the number of sampled votes v, from to 5n(n ) and vary worker accuracy p from.55 to.95. As a point of reference, we refer to n(n ) votes as v = x Edge Coverage, e.g. each pair of objects is sampled approximately once. So 5n(n ) votes is equivalent to v = x Edge Coverage in our experiments. Each data point (given n, p, v values) in our results graphs is obtained from 5, runs. Each run proceeds as follows: We initialize W as an n n null matrix and begin with an arbitrary true permutation π of the objects in O. Let U denote the set of all tuples (i, j) where i j. We randomly sample v tuples from U with replacement. After sampling a tuple (i, j), we simulate the human worker s comparison of objects o i and o j. If π (i) < π (j), with probability p, we increment w ji, and with probability p, we increment w ij. If π (j) < π (i), with probability p, we increment w ij, and with probability p, we increment w ji. For each generated matrix W in a run, we apply each of our heuristic strategies to obtain predicted rankings of the objects in O. Comparing the predicted ranking with π we record both (a) a yes if the predicted maximum agrees with the true maximum, and (b) reciprocal rank, the inverse rank of the true maximum object in the predicted ranking. Finally, after all runs complete, we compute (a) Precision at the fraction of yes cases over the number of runs, and (b) the Mean Reciprocal Rank (MRR), the average reciprocal rank over all runs. As a first experiment, we consider the prediction performance of Maximum Likelihood (ML) and the four heuristics for a set of 5 objects with p =.75, displayed in Figure. We choose a small set of objects, so that ML can be computed. Looking at Figure (left), we find that as the number of votes sampled increases, the of all heuristics (excluding PageRank) increase in a concave manner, approaching a value of.9 for x Edge Coverage. In other words, if 5n(n ) votes are uniformly sampled, the heuristics can predict the maximum object 9% of the time, even though average worker accuracy is.75. Similar prediction curves measuring MRR are displayed in Figure (right). ML has better performance than all the four heuristics. As expected, ML performs the best in Figure, but recall that ML requires explicit knowledge of p, and it is computationally very expensive. Still, the ML curve is useful, since it tells us how far the heuristics are from the optimal feasible solution (ML). Also, note that PageRank () performs poorly in Figure, indicating that PageRank is poor when the number of objects is small. Iterative is the best of the four heuristics when the number of votes sampled is n(n ), e.g. x Edge Coverage. For a larger experiment, we consider the problem of prediction for n = objects in Figure 3. ML is necessarily omitted from this experiment. Looking at the graphs, we first note that the Iterative () heuristic performs significantly better than the other heuristics, particularly when p =.55 or p =.75. This is best demonstrated by Figure 3(middle), which shows that for p =.75 and x Edge Coverage, the Iterative heuristic has a of over.9, whereas the second best heuristic, Indegree (), only has a of approximately.5. Looking at the middle graph again, note how the performance gap between the Iterative heuristic and

12 to compare the top-ranked object against other objects greedily. For example, with a budget of b =, this strategy asks human workers to compare the rank and rank objects, and the rank and rank 3 objects, where rank is determined by the scoring function in Step in Algorithm 6. Considering again the example in Figure 6, for b =, this strategy requests the votes (A, B) and (A, E). Strategy 9 Greedy Vote Selection Require: n objects, budget b, vote multiset Q, scores s[ ], sorted object indices index[ ] Ensure: Q = selected b votes initialize priority queue S {storing unordered object pairs} for i :... b do for j : (i + )... b do insert object pair (i, j) into S with priority (s[i] s[j]) for i :... b do Remove highest priority object pair (x, y) from S Q Q (x,y) The third strategy we consider is Greedy vote selection (), displayed in Strategy 9. In this strategy, all possible comparisons (unordered object pairs) are weighted by the product of the scores of the two objects, where the scores are determined in Step of Algorithm 6. In other words, a distribution is constructed across all possible object pairs, with higher weights assigned to object pairs involving high scoring objects (which are more likely to be the maximum object in O). After weighting all possible object pairs, this strategy submits the b highest weight pairs for human comparison. Considering the example in Figure 6, object pairs (A, B) and (A, E) has weight.5, (B, E) has weight.65, and all other pairs have weight. For a budget b =, this strategy requests the votes (A, B) and (A,E). Strategy Complete (Round-Robin) Vote Selection Require: n objects, budget b, vote multiset Q, scores s[ ], sorted object indices index[ ] Ensure: Q = selected b votes K {K is the size of the round-robin tournament} while K (K+) b do K K + end while K K for i :... K do for j : (i + )... K do Q Q (index[i], index[j]) {index[] has largest score s} initialize priority queue S {storing unordered object pairs} for i :... K do insert object pair (i, K +) into S with priority (s[i] s[k + ]) for i :...(b K (K+) ) do Remove highest priority object pair (x, y) from S Q Q (x, y) {select remaining votes greedily} The fourth strategy we consider is Complete Tournament vote selection (), displayed in Strategy. In this strategy, Precision at ML-ML ML-.8.6 MA Figure 7: Precision at versus number of initial votes. additional vote, 7 objects, p=.75. we construct a single round-robin tournament among the K objects with the highest scores from Step of Algorithm 6, where K is the largest number such that K (K+) b. In a single roundrobin tournament, each of the K objects is compared against every other exactly once. For the remaining r = b K (K+) votes, we consider all object pairs containing the (K + )st (largest scoring) object and one of the first K objects, and weight each of these K object pairs by the product of the scores of the two objects (as we did with Greedy vote selection). We then select the r object pairs with highest weight. The idea behind the Complete Tournament strategy is that a roundrobin tournament will likely determine the largest object among the set of K objects. If the set of K objects contains the true max, this strategy can be anticipated to perform well. Regarding the selection of the remaining votes, the strategy can be thought of as augmenting the K object tournament to become an incomplete K + object tournament, where the remaining votes are selected greedily to best determine if the (K+)st object can possibly be the maximum object in O. Considering the example in Figure 6, for b =, there is a -object tournament among objects A and B and vote (A, B) is requested. Then, for the remaining vote, the strategy greedily scores object pairs which contain both the next highest ranked object not in the tournament, object E, and one of the initial objects. Object pair (A, E) will be scored.5 and (B, E) will be scored.65, so the second vote requested is (A, E). 3.4 Experiments Which of our four vote selection heuristics (, MA,, or ) is the best strategy? We now describe a set of experiments measuring the prediction performance of our heuristics for various sets of parameters. When evaluating our vote selection strategies, we utilized a uniform vote sampling procedure, described previously in Section.5, to generate an initial vote matrix W. Then, in Step of our vote selection framework (Algorithm 6), we adopted our PageRank heuristic (Section.4) as our scoring function s( ) to score each object in O. In Step, we executed each of our vote selection strategies using these scores. In Step 3, we used our PageRank heuristic as our scoring function f( ) to score each object in the new matrix W (composed of both the initial votes in vote matrix W and the b requested additional votes), and generate final predictions for the maximum object in O. Note that we performed several experiments contrasting prediction performance of PageRank versus other possible scoring functions and found PageRank to be superior to the other functions. Hence, we selected PageRank as the scoring function for both Step and Step 3 of our vote selection framework.

15 D are both admissible solutions for the Kemeny rule, ML returns D as an answer, since D has almost one and a half more times the probability of being the maximum object compared to C. No prior work about the Judgment Problem, to our knowledge, uses the same approach as our ML formulation. In the recent social choice literature, the research most closely related to ours has been work by Conitzer et al. regarding Kemeny permutations and Maximum Likelihood [9, ]. Conitzer has studied various voting rules and determined for which of them there exist voter error models where the rules are ML estimators [8]. In our study, we focused upon the opposite question: for a specific voter error model, we presented both Maximum Likelihood, as well as heuristic solutions, to predict the winner. Our work is also related to research in the theory community regarding ranking in the presence of errors [, ] and noisy computation [4, ]. Both Kenyon et al. and Ailon et al. present randomized polynomial-time algorithms for feedback arc set in tournament graphs. Their algorithms are intended to approximate the optimal permutation, whereas we seek to predict the optimal winner. Feige et al. and Ajtai et al. present algorithms to solve a variety of problems, including the Max Problem, but their scenarios involve different comparison models or error models than ours. More generally, in the last several years, there has been a significant amount of work regarding crowdsourcing systems, both inside [5, 6] and outside [, 3] the database community. Of note is recent work by Tamuz et al. [3] on a crowdsourcing system that learns a similarity matrix across objects, while adaptively requesting votes. Not as much work has been done regarding general crowdsourcing algorithms [4, 7]. Instead, most algorithmic work in crowdsourcing has focused upon quality control [3, 8]. 5. CONCLUSION In a conventional database system, finding the maximum element in a set is a relatively simple procedure. It is thus somewhat surprising that in a crowdsourcing database system, finding the maximum is quite challenging, and there are many issues to consider. The main reason for the complexity, as we have seen, is that our underlying comparison operation may give an incorrect answer, or it may even not complete. Thus, we need to decide which is the most likely max (Judgment Problem), and which additional votes to request to improve our answer (Next Votes Problem). Our results show that solving either one of these problems optimally is very hard, but fortunately we have proposed effective heuristics that do well. There is a gap between the optimal solution and what our heuristics find (as seen for example in Figure 7), but we believe that it will be very hard to close this gap without incurring high computational costs. Among the heuristics, we observed significant differences in their predictive ability, indicating that it is very important to carefully select a good heuristic. Our results indicate that in many cases (but not all) our proposed PageRank heuristic is the best for the Judgment Problem, while the Complete Tournament heuristic is the best for the Next Votes Problem. Our results are based on a relatively simple model where object comparisons are pairwise, and worker errors are independent. Of course, in a real crowdsourced database system these assumptions may not hold. Yet we believe it is important to know that even with the simple model, the optimal strategies for the Judgment Problem and Next Votes Problem are NP-Hard. Furthermore, our heuristics can be used even in more complex scenarios, since they do not depend on the evaluation model. Even though they can be used when the model assumptions do not hold, we believe it is important to understand how the heuristics perform in the more tractable scenario we have considered here. 6. REFERENCES [] N. Ailon et al. Aggregating inconsistent information: ranking and clustering. JACM, 55(5), 8. [] M. Ajtai et al. Sorting and selection with imprecise comparisons. Automata, Languages and Programming, 9. [3] O. Alonso et al. Crowdsourcing for relevance evaluation. In SIGIR Forum, 8. [4] P. Boldi et al. PageRank as a function of the damping factor. In WWW, 5. [5] F. Brandt et al. PageRank as a weak tournament solution. In WINE, 7. [6] Y. Chevaleyre et al. A short introduction to computational social choice. In SOFSEM, 7. [7] W. Cohen et al. Learning to order things. Journal of Artificial Intelligence Research, :43 7, 999. [8] V. Conitzer et al. Common voting rules as maximum likelihood estimators. In UAI, 5. [9] V. Conitzer et al. Improved bounds for computing kemeny rankings. In AAAI, 6. [] V. Conitzer et al. Preference functions that score rankings and maximum likelihood estimation. In IJCAI, 9. [] D. Coppersmith et al. Ordering by weighted number of wins gives a good ranking for weighted tournaments. In SODA, 6. [] H.A. David. The method of paired comparisons [3] H.A. David. Ranking from unbalanced paired-comparison data. Biometrika, 74():43 436, 987. [4] U. Feige et al. Computing with noisy information. SIAM J. Comput., 3(5): 8, 994. [5] M. Franklin et al. CrowdDB: answering queries with crowdsourcing. In SIGMOD,. [6] P. Heymann et al. Turkalytics: analytics for human computation. In WWW,. [7] O. Hudry. On the complexity of Slater s problems. European Journal of Operational Research, 3():6,. [8] P. Ipeirotis et al. Quality management on amazon mechanical turk. In KDD Workshop on Human Computation,. [9] J. Kemeny. Mathematics without numbers. Daedalus, 959. [] M.G. Kendall et al. On the method of paired comparisons. Biometrika, 3(3/4):34 345, 94. [] C. Kenyon-Mathieu et al. How to rank with few errors. In STOC, 7. [] A. Kittur et al. Crowdsourcing user studies with mechanical turk. In CHI, 8. [3] G. Little et al. TurKit: human computation algorithms on mechanical turk. In UIST,. [4] A. Marcus et al. Human-powered sorts and joins. PVLDB,. [5] H. Moulin. Choosing from a tournament. Social Choice and Welfare, 3(4):7 9, 986. [6] L. Page et al. The PageRank citation ranking: Bringing order to the web. Technical Report, 998. [7] A. Parameswaran et al. Human-assisted graph search: it s okay to ask questions. PVLDB,. [8] D. Saari. Basic geometry of voting [9] P. Slater. Inconsistencies in a schedule of paired comparisons. Biometrika, 48(3/4):33 3, 96. [3] O. Tamuz et al. Adaptively learning the crowd kernel. In ICML,. [3] P. Young. Optimal voting rules. J. Econ. Perspectives, 995.

### Globally Optimal Crowdsourcing Quality Management

Globally Optimal Crowdsourcing Quality Management Akash Das Sarma Stanford University akashds@stanford.edu Aditya G. Parameswaran University of Illinois (UIUC) adityagp@illinois.edu Jennifer Widom Stanford

### December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B. KITCHENS

December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B KITCHENS The equation 1 Lines in two-dimensional space (1) 2x y = 3 describes a line in two-dimensional space The coefficients of x and y in the equation

### Distributed Computing over Communication Networks: Maximal Independent Set

Distributed Computing over Communication Networks: Maximal Independent Set What is a MIS? MIS An independent set (IS) of an undirected graph is a subset U of nodes such that no two nodes in U are adjacent.

### Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

### Chapter 21: The Discounted Utility Model

Chapter 21: The Discounted Utility Model 21.1: Introduction This is an important chapter in that it introduces, and explores the implications of, an empirically relevant utility function representing intertemporal

### CHAPTER 2 Estimating Probabilities

CHAPTER 2 Estimating Probabilities Machine Learning Copyright c 2016. Tom M. Mitchell. All rights reserved. *DRAFT OF January 24, 2016* *PLEASE DO NOT DISTRIBUTE WITHOUT AUTHOR S PERMISSION* This is a

### Minimize subject to. x S R

Chapter 12 Lagrangian Relaxation This chapter is mostly inspired by Chapter 16 of [1]. In the previous chapters, we have succeeded to find efficient algorithms to solve several important problems such

### Part 2: Community Detection

Chapter 8: Graph Data Part 2: Community Detection Based on Leskovec, Rajaraman, Ullman 2014: Mining of Massive Datasets Big Data Management and Analytics Outline Community Detection - Social networks -

### 26. Determinants I. 1. Prehistory

26. Determinants I 26.1 Prehistory 26.2 Definitions 26.3 Uniqueness and other properties 26.4 Existence Both as a careful review of a more pedestrian viewpoint, and as a transition to a coordinate-independent

### The concept of probability is fundamental in statistical analysis. Theory of probability underpins most of the methods used in statistics.

Elementary probability theory The concept of probability is fundamental in statistical analysis. Theory of probability underpins most of the methods used in statistics. 1.1 Experiments, outcomes and sample

### Gerry Hobbs, Department of Statistics, West Virginia University

Decision Trees as a Predictive Modeling Method Gerry Hobbs, Department of Statistics, West Virginia University Abstract Predictive modeling has become an important area of interest in tasks such as credit

### Notes from February 11

Notes from February 11 Math 130 Course web site: www.courses.fas.harvard.edu/5811 Two lemmas Before proving the theorem which was stated at the end of class on February 8, we begin with two lemmas. The

### Approximation Algorithms

Approximation Algorithms or: How I Learned to Stop Worrying and Deal with NP-Completeness Ong Jit Sheng, Jonathan (A0073924B) March, 2012 Overview Key Results (I) General techniques: Greedy algorithms

### Random graphs with a given degree sequence

Sourav Chatterjee (NYU) Persi Diaconis (Stanford) Allan Sly (Microsoft) Let G be an undirected simple graph on n vertices. Let d 1,..., d n be the degrees of the vertices of G arranged in descending order.

### Cost Model: Work, Span and Parallelism. 1 The RAM model for sequential computation:

CSE341T 08/31/2015 Lecture 3 Cost Model: Work, Span and Parallelism In this lecture, we will look at how one analyze a parallel program written using Cilk Plus. When we analyze the cost of an algorithm

### Introduction to Flocking {Stochastic Matrices}

Supelec EECI Graduate School in Control Introduction to Flocking {Stochastic Matrices} A. S. Morse Yale University Gif sur - Yvette May 21, 2012 CRAIG REYNOLDS - 1987 BOIDS The Lion King CRAIG REYNOLDS

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

### The PageRank Citation Ranking: Bring Order to the Web

The PageRank Citation Ranking: Bring Order to the Web presented by: Xiaoxi Pang 25.Nov 2010 1 / 20 Outline Introduction A ranking for every page on the Web Implementation Convergence Properties Personalized

### 6.042/18.062J Mathematics for Computer Science October 3, 2006 Tom Leighton and Ronitt Rubinfeld. Graph Theory III

6.04/8.06J Mathematics for Computer Science October 3, 006 Tom Leighton and Ronitt Rubinfeld Lecture Notes Graph Theory III Draft: please check back in a couple of days for a modified version of these

### 1 Limiting distribution for a Markov chain

Copyright c 2009 by Karl Sigman Limiting distribution for a Markov chain In these Lecture Notes, we shall study the limiting behavior of Markov chains as time n In particular, under suitable easy-to-check

### UNIT 2 MATRICES - I 2.0 INTRODUCTION. Structure

UNIT 2 MATRICES - I Matrices - I Structure 2.0 Introduction 2.1 Objectives 2.2 Matrices 2.3 Operation on Matrices 2.4 Invertible Matrices 2.5 Systems of Linear Equations 2.6 Answers to Check Your Progress

### Permutation Betting Markets: Singleton Betting with Extra Information

Permutation Betting Markets: Singleton Betting with Extra Information Mohammad Ghodsi Sharif University of Technology ghodsi@sharif.edu Hamid Mahini Sharif University of Technology mahini@ce.sharif.edu

### CORRELATED TO THE SOUTH CAROLINA COLLEGE AND CAREER-READY FOUNDATIONS IN ALGEBRA

We Can Early Learning Curriculum PreK Grades 8 12 INSIDE ALGEBRA, GRADES 8 12 CORRELATED TO THE SOUTH CAROLINA COLLEGE AND CAREER-READY FOUNDATIONS IN ALGEBRA April 2016 www.voyagersopris.com Mathematical

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. !-approximation algorithm.

Approximation Algorithms Chapter Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of

### Lecture 15 An Arithmetic Circuit Lowerbound and Flows in Graphs

CSE599s: Extremal Combinatorics November 21, 2011 Lecture 15 An Arithmetic Circuit Lowerbound and Flows in Graphs Lecturer: Anup Rao 1 An Arithmetic Circuit Lower Bound An arithmetic circuit is just like

### CSC2420 Fall 2012: Algorithm Design, Analysis and Theory

CSC2420 Fall 2012: Algorithm Design, Analysis and Theory Allan Borodin November 15, 2012; Lecture 10 1 / 27 Randomized online bipartite matching and the adwords problem. We briefly return to online algorithms

### Network (Tree) Topology Inference Based on Prüfer Sequence

Network (Tree) Topology Inference Based on Prüfer Sequence C. Vanniarajan and Kamala Krithivasan Department of Computer Science and Engineering Indian Institute of Technology Madras Chennai 600036 vanniarajanc@hcl.in,

### Principle of Data Reduction

Chapter 6 Principle of Data Reduction 6.1 Introduction An experimenter uses the information in a sample X 1,..., X n to make inferences about an unknown parameter θ. If the sample size n is large, then

### On the Interaction and Competition among Internet Service Providers

On the Interaction and Competition among Internet Service Providers Sam C.M. Lee John C.S. Lui + Abstract The current Internet architecture comprises of different privately owned Internet service providers

### Math 312 Homework 1 Solutions

Math 31 Homework 1 Solutions Last modified: July 15, 01 This homework is due on Thursday, July 1th, 01 at 1:10pm Please turn it in during class, or in my mailbox in the main math office (next to 4W1) Please

### Lecture 3: Linear Programming Relaxations and Rounding

Lecture 3: Linear Programming Relaxations and Rounding 1 Approximation Algorithms and Linear Relaxations For the time being, suppose we have a minimization problem. Many times, the problem at hand can

### OPRE 6201 : 2. Simplex Method

OPRE 6201 : 2. Simplex Method 1 The Graphical Method: An Example Consider the following linear program: Max 4x 1 +3x 2 Subject to: 2x 1 +3x 2 6 (1) 3x 1 +2x 2 3 (2) 2x 2 5 (3) 2x 1 +x 2 4 (4) x 1, x 2

### 1 Introductory Comments. 2 Bayesian Probability

Introductory Comments First, I would like to point out that I got this material from two sources: The first was a page from Paul Graham s website at www.paulgraham.com/ffb.html, and the second was a paper

### Crosswalk Directions:

Crosswalk Directions: UMS Standards for College Readiness to 2007 MLR 1. Use a (yes), an (no), or a (partially) to indicate the extent to which the standard, performance indicator, or descriptor of the

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### pp. 4 8: Examples 1 6 Quick Check 1 6 Exercises 1, 2, 20, 42, 43, 64

Semester 1 Text: Chapter 1: Tools of Algebra Lesson 1-1: Properties of Real Numbers Day 1 Part 1: Graphing and Ordering Real Numbers Part 1: Graphing and Ordering Real Numbers Lesson 1-2: Algebraic Expressions

### Procedia Computer Science 00 (2012) 1 21. Trieu Minh Nhut Le, Jinli Cao, and Zhen He. trieule@sgu.edu.vn, j.cao@latrobe.edu.au, z.he@latrobe.edu.

Procedia Computer Science 00 (2012) 1 21 Procedia Computer Science Top-k best probability queries and semantics ranking properties on probabilistic databases Trieu Minh Nhut Le, Jinli Cao, and Zhen He

### South Carolina College- and Career-Ready (SCCCR) Probability and Statistics

South Carolina College- and Career-Ready (SCCCR) Probability and Statistics South Carolina College- and Career-Ready Mathematical Process Standards The South Carolina College- and Career-Ready (SCCCR)

### The Basics of Graphical Models

The Basics of Graphical Models David M. Blei Columbia University October 3, 2015 Introduction These notes follow Chapter 2 of An Introduction to Probabilistic Graphical Models by Michael Jordan. Many figures

### Relations Graphical View

Relations Slides by Christopher M. Bourke Instructor: Berthe Y. Choueiry Introduction Recall that a relation between elements of two sets is a subset of their Cartesian product (of ordered pairs). A binary

### Chapter 2: Systems of Linear Equations and Matrices:

At the end of the lesson, you should be able to: Chapter 2: Systems of Linear Equations and Matrices: 2.1: Solutions of Linear Systems by the Echelon Method Define linear systems, unique solution, inconsistent,

### Min-cost flow problems and network simplex algorithm

Min-cost flow problems and network simplex algorithm The particular structure of some LP problems can be sometimes used for the design of solution techniques more efficient than the simplex algorithm.

### Practical Graph Mining with R. 5. Link Analysis

Practical Graph Mining with R 5. Link Analysis Outline Link Analysis Concepts Metrics for Analyzing Networks PageRank HITS Link Prediction 2 Link Analysis Concepts Link A relationship between two entities

PYTHAGOREAN TRIPLES KEITH CONRAD 1. Introduction A Pythagorean triple is a triple of positive integers (a, b, c) where a + b = c. Examples include (3, 4, 5), (5, 1, 13), and (8, 15, 17). Below is an ancient

### Exponential time algorithms for graph coloring

Exponential time algorithms for graph coloring Uriel Feige Lecture notes, March 14, 2011 1 Introduction Let [n] denote the set {1,..., k}. A k-labeling of vertices of a graph G(V, E) is a function V [k].

### On the Efficiency of Competitive Stock Markets Where Traders Have Diverse Information

Finance 400 A. Penati - G. Pennacchi Notes on On the Efficiency of Competitive Stock Markets Where Traders Have Diverse Information by Sanford Grossman This model shows how the heterogeneous information

### Discrete Mathematics & Mathematical Reasoning Chapter 10: Graphs

Discrete Mathematics & Mathematical Reasoning Chapter 10: Graphs Kousha Etessami U. of Edinburgh, UK Kousha Etessami (U. of Edinburgh, UK) Discrete Mathematics (Chapter 6) 1 / 13 Overview Graphs and Graph

### Spring 2007 Math 510 Hints for practice problems

Spring 2007 Math 510 Hints for practice problems Section 1 Imagine a prison consisting of 4 cells arranged like the squares of an -chessboard There are doors between all adjacent cells A prisoner in one

### INTRODUCTION TO PROOFS: HOMEWORK SOLUTIONS

INTRODUCTION TO PROOFS: HOMEWORK SOLUTIONS STEVEN HEILMAN Contents 1. Homework 1 1 2. Homework 2 6 3. Homework 3 10 4. Homework 4 16 5. Homework 5 19 6. Homework 6 21 7. Homework 7 25 8. Homework 8 28

### MATH 304 Linear Algebra Lecture 4: Matrix multiplication. Diagonal matrices. Inverse matrix.

MATH 304 Linear Algebra Lecture 4: Matrix multiplication. Diagonal matrices. Inverse matrix. Matrices Definition. An m-by-n matrix is a rectangular array of numbers that has m rows and n columns: a 11

### Overview of Math Standards

Algebra 2 Welcome to math curriculum design maps for Manhattan- Ogden USD 383, striving to produce learners who are: Effective Communicators who clearly express ideas and effectively communicate with diverse

### Generating Labels from Clicks

Generating Labels from Clicks R. Agrawal A. Halverson K. Kenthapadi N. Mishra P. Tsaparas Search Labs, Microsoft Research {rakesha,alanhal,krisken,ninam,panats}@microsoft.com ABSTRACT The ranking function

### Social Media Mining. Graph Essentials

Graph Essentials Graph Basics Measures Graph and Essentials Metrics 2 2 Nodes and Edges A network is a graph nodes, actors, or vertices (plural of vertex) Connections, edges or ties Edge Node Measures

Approximation Algorithms Chapter Approximation Algorithms Q. Suppose I need to solve an NP-hard problem. What should I do? A. Theory says you're unlikely to find a poly-time algorithm. Must sacrifice one

### Load Balancing and Switch Scheduling

EE384Y Project Final Report Load Balancing and Switch Scheduling Xiangheng Liu Department of Electrical Engineering Stanford University, Stanford CA 94305 Email: liuxh@systems.stanford.edu Abstract Load

### Offline sorting buffers on Line

Offline sorting buffers on Line Rohit Khandekar 1 and Vinayaka Pandit 2 1 University of Waterloo, ON, Canada. email: rkhandekar@gmail.com 2 IBM India Research Lab, New Delhi. email: pvinayak@in.ibm.com

### LOOKING FOR A GOOD TIME TO BET

LOOKING FOR A GOOD TIME TO BET LAURENT SERLET Abstract. Suppose that the cards of a well shuffled deck of cards are turned up one after another. At any time-but once only- you may bet that the next card

### MINITAB ASSISTANT WHITE PAPER

MINITAB ASSISTANT WHITE PAPER This paper explains the research conducted by Minitab statisticians to develop the methods and data checks used in the Assistant in Minitab 17 Statistical Software. One-Way

### ORDERS OF ELEMENTS IN A GROUP

ORDERS OF ELEMENTS IN A GROUP KEITH CONRAD 1. Introduction Let G be a group and g G. We say g has finite order if g n = e for some positive integer n. For example, 1 and i have finite order in C, since

### NCSS Statistical Software Principal Components Regression. In ordinary least squares, the regression coefficients are estimated using the formula ( )

Chapter 340 Principal Components Regression Introduction is a technique for analyzing multiple regression data that suffer from multicollinearity. When multicollinearity occurs, least squares estimates

### 1. R In this and the next section we are going to study the properties of sequences of real numbers.

+a 1. R In this and the next section we are going to study the properties of sequences of real numbers. Definition 1.1. (Sequence) A sequence is a function with domain N. Example 1.2. A sequence of real

### Chapter 6: Episode discovery process

Chapter 6: Episode discovery process Algorithmic Methods of Data Mining, Fall 2005, Chapter 6: Episode discovery process 1 6. Episode discovery process The knowledge discovery process KDD process of analyzing

### 2.3 Scheduling jobs on identical parallel machines

2.3 Scheduling jobs on identical parallel machines There are jobs to be processed, and there are identical machines (running in parallel) to which each job may be assigned Each job = 1,,, must be processed

### Social Media Mining. Network Measures

Klout Measures and Metrics 22 Why Do We Need Measures? Who are the central figures (influential individuals) in the network? What interaction patterns are common in friends? Who are the like-minded users

### Notes on Determinant

ENGG2012B Advanced Engineering Mathematics Notes on Determinant Lecturer: Kenneth Shum Lecture 9-18/02/2013 The determinant of a system of linear equations determines whether the solution is unique, without

### You know from calculus that functions play a fundamental role in mathematics.

CHPTER 12 Functions You know from calculus that functions play a fundamental role in mathematics. You likely view a function as a kind of formula that describes a relationship between two (or more) quantities.

### Math Review. for the Quantitative Reasoning Measure of the GRE revised General Test

Math Review for the Quantitative Reasoning Measure of the GRE revised General Test www.ets.org Overview This Math Review will familiarize you with the mathematical skills and concepts that are important

Mechanisms for Fair Attribution Eric Balkanski Yaron Singer Abstract We propose a new framework for optimization under fairness constraints. The problems we consider model procurement where the goal is

### 17.6.1 Introduction to Auction Design

CS787: Advanced Algorithms Topic: Sponsored Search Auction Design Presenter(s): Nilay, Srikrishna, Taedong 17.6.1 Introduction to Auction Design The Internet, which started of as a research project in

### SCHOOL DISTRICT OF THE CHATHAMS CURRICULUM PROFILE

CONTENT AREA(S): Mathematics COURSE/GRADE LEVEL(S): Honors Algebra 2 (10/11) I. Course Overview In Honors Algebra 2, the concept of mathematical function is developed and refined through the study of real

### THREE DIMENSIONAL GEOMETRY

Chapter 8 THREE DIMENSIONAL GEOMETRY 8.1 Introduction In this chapter we present a vector algebra approach to three dimensional geometry. The aim is to present standard properties of lines and planes,

### Effective Page Refresh Policies For Web Crawlers

Effective Page Refresh Policies For Web Crawlers JUNGHOO CHO University of California, Los Angeles and HECTOR GARCIA-MOLINA Stanford University In this paper we study how we can maintain local copies of

### Mathematical Procedures

CHAPTER 6 Mathematical Procedures 168 CHAPTER 6 Mathematical Procedures The multidisciplinary approach to medicine has incorporated a wide variety of mathematical procedures from the fields of physics,

### Matrices: 2.3 The Inverse of Matrices

September 4 Goals Define inverse of a matrix. Point out that not every matrix A has an inverse. Discuss uniqueness of inverse of a matrix A. Discuss methods of computing inverses, particularly by row operations.

### Linear Programming I

Linear Programming I November 30, 2003 1 Introduction In the VCR/guns/nuclear bombs/napkins/star wars/professors/butter/mice problem, the benevolent dictator, Bigus Piguinus, of south Antarctica penguins

### Discrete Mathematics and Probability Theory Fall 2009 Satish Rao,David Tse Note 11

CS 70 Discrete Mathematics and Probability Theory Fall 2009 Satish Rao,David Tse Note Conditional Probability A pharmaceutical company is marketing a new test for a certain medical condition. According

### Mining Social-Network Graphs

342 Chapter 10 Mining Social-Network Graphs There is much information to be gained by analyzing the large-scale data that is derived from social networks. The best-known example of a social network is

### ON DEGREES IN THE HASSE DIAGRAM OF THE STRONG BRUHAT ORDER

Séminaire Lotharingien de Combinatoire 53 (2006), Article B53g ON DEGREES IN THE HASSE DIAGRAM OF THE STRONG BRUHAT ORDER RON M. ADIN AND YUVAL ROICHMAN Abstract. For a permutation π in the symmetric group

Section 5.4 The Quadratic Formula 481 5.4 The Quadratic Formula Consider the general quadratic function f(x) = ax + bx + c. In the previous section, we learned that we can find the zeros of this function

### Matrix Algebra LECTURE 1. Simultaneous Equations Consider a system of m linear equations in n unknowns: y 1 = a 11 x 1 + a 12 x 2 + +a 1n x n,

LECTURE 1 Matrix Algebra Simultaneous Equations Consider a system of m linear equations in n unknowns: y 1 a 11 x 1 + a 12 x 2 + +a 1n x n, (1) y 2 a 21 x 1 + a 22 x 2 + +a 2n x n, y m a m1 x 1 +a m2 x

### Minimizing Probing Cost and Achieving Identifiability in Probe Based Network Link Monitoring

Minimizing Probing Cost and Achieving Identifiability in Probe Based Network Link Monitoring Qiang Zheng, Student Member, IEEE, and Guohong Cao, Fellow, IEEE Department of Computer Science and Engineering

### Mathematics for Computer Science/Software Engineering. Notes for the course MSM1F3 Dr. R. A. Wilson

Mathematics for Computer Science/Software Engineering Notes for the course MSM1F3 Dr. R. A. Wilson October 1996 Chapter 1 Logic Lecture no. 1. We introduce the concept of a proposition, which is a statement

### Corollary. (f є C n+1 [a,b]). Proof: This follows directly from the preceding theorem using the inequality

Corollary For equidistant knots, i.e., u i = a + i (b-a)/n, we obtain with (f є C n+1 [a,b]). Proof: This follows directly from the preceding theorem using the inequality 120202: ESM4A - Numerical Methods

Load Balancing Backtracking, branch & bound and alpha-beta pruning: how to assign work to idle processes without much communication? Additionally for alpha-beta pruning: implementing the young-brothers-wait

### Moral Hazard. Itay Goldstein. Wharton School, University of Pennsylvania

Moral Hazard Itay Goldstein Wharton School, University of Pennsylvania 1 Principal-Agent Problem Basic problem in corporate finance: separation of ownership and control: o The owners of the firm are typically

### FUZZY CLUSTERING ANALYSIS OF DATA MINING: APPLICATION TO AN ACCIDENT MINING SYSTEM

International Journal of Innovative Computing, Information and Control ICIC International c 0 ISSN 34-48 Volume 8, Number 8, August 0 pp. 4 FUZZY CLUSTERING ANALYSIS OF DATA MINING: APPLICATION TO AN ACCIDENT

### princeton univ. F 13 cos 521: Advanced Algorithm Design Lecture 6: Provable Approximation via Linear Programming Lecturer: Sanjeev Arora

princeton univ. F 13 cos 521: Advanced Algorithm Design Lecture 6: Provable Approximation via Linear Programming Lecturer: Sanjeev Arora Scribe: One of the running themes in this course is the notion of

### Continued Fractions and the Euclidean Algorithm

Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction

### U.C. Berkeley CS276: Cryptography Handout 0.1 Luca Trevisan January, 2009. Notes on Algebra

U.C. Berkeley CS276: Cryptography Handout 0.1 Luca Trevisan January, 2009 Notes on Algebra These notes contain as little theory as possible, and most results are stated without proof. Any introductory

### 10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES

58 CHAPTER NUMERICAL METHODS. POWER METHOD FOR APPROXIMATING EIGENVALUES In Chapter 7 you saw that the eigenvalues of an n n matrix A are obtained by solving its characteristic equation n c nn c nn...

### Elementary Number Theory We begin with a bit of elementary number theory, which is concerned

CONSTRUCTION OF THE FINITE FIELDS Z p S. R. DOTY Elementary Number Theory We begin with a bit of elementary number theory, which is concerned solely with questions about the set of integers Z = {0, ±1,

### Alok Gupta. Dmitry Zhdanov

RESEARCH ARTICLE GROWTH AND SUSTAINABILITY OF MANAGED SECURITY SERVICES NETWORKS: AN ECONOMIC PERSPECTIVE Alok Gupta Department of Information and Decision Sciences, Carlson School of Management, University

### CMSC 858T: Randomized Algorithms Spring 2003 Handout 8: The Local Lemma

CMSC 858T: Randomized Algorithms Spring 2003 Handout 8: The Local Lemma Please Note: The references at the end are given for extra reading if you are interested in exploring these ideas further. You are

### Mathematics Review for MS Finance Students

Mathematics Review for MS Finance Students Anthony M. Marino Department of Finance and Business Economics Marshall School of Business Lecture 1: Introductory Material Sets The Real Number System Functions,

### 3.3. Solving Polynomial Equations. Introduction. Prerequisites. Learning Outcomes

Solving Polynomial Equations 3.3 Introduction Linear and quadratic equations, dealt within Sections 3.1 and 3.2, are members of a class of equations, called polynomial equations. These have the general

### Information Security and Risk Management

Information Security and Risk Management by Lawrence D. Bodin Professor Emeritus of Decision and Information Technology Robert H. Smith School of Business University of Maryland College Park, MD 20742

### Markov random fields and Gibbs measures

Chapter Markov random fields and Gibbs measures 1. Conditional independence Suppose X i is a random element of (X i, B i ), for i = 1, 2, 3, with all X i defined on the same probability space (.F, P).