Random Walk Inference and Learning in A Large Scale Knowledge Base


 Imogene Lang
 1 years ago
 Views:
Transcription
1 Random Walk Inferene and Learning in A Large Sale Knowledge Base Ni Lao Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA Tom Mithell Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA William W. Cohen Carnegie Mellon University 5000 Forbes Avenue Pittsburgh, PA Abstrat We onsider the problem of performing learning and inferene in a large sale knowledge base ontaining imperfet knowledge with inomplete overage. We show that a soft inferene proedure based on a ombination of onstrained, weighted, random walks through the knowledge base graph an be used to reliably infer new beliefs for the knowledge base. More speifially, we show that the system an learn to infer different target relations by tuning the weights assoiated with random walks that follow different paths through the graph, using a version of the Path Ranking Algorithm (Lao and Cohen, 2010b). We apply this approah to a knowledge base of approximately 500,000 beliefs extrated imperfetly from the web by NELL, a neverending language learner (Carlson et al., 2010). This new system improves signifiantly over NELL s earlier Hornlause learning and inferene method: it obtains nearly double the preision at rank 100, and the new learning method is also appliable to many more inferene tasks. 1 Introdution Although there is a great deal of reent researh on extrating knowledge from text (Agihtein and Gravano, 2000; Etzioni et al., 2005; Snow et al., 2006; Pantel and Pennahiotti, 2006; Banko et al., 2007; Yates et al., 2007), muh less progress has been made on the problem of drawing reliable inferenes from this imperfetly extrated knowledge. In partiular, traditional logial inferene methods are too brittle to be used to make omplex inferenes from automatiallyextrated knowledge, and probabilisti inferene methods (Rihardson and Domingos, 2006) suffer from salability problems. This paper onsiders the problem of onstruting inferene methods that an sale to large KB s, and that are robust to imperfet knowledge. The KB we onsider is a large triple store, whih an be represented as a labeled, direted graph in whih eah entity x is a node, eah binary relation R(x, y) is an edge labeled R between x and y, and unary onepts C(x) are represented as an edge labeled isa between the node for the entity x and a node for the onept C. We present a trainable inferene method that learns to infer relations by ombining the results of different random walks through this graph, and show that the method ahieves good saling properties and robust inferene in a KB ontaining over 500,000 triples extrated from the web by the NELL system (Carlson et al., 2010). 1.1 The NELL Case Study To evaluate our approah experimentally, we study it in the ontext of the NELL (Never Ending Language Learning) researh projet, whih is an effort to develop a neverending learning system that operates 24 hours per day, for years, to ontinuously improve its ability to read (extrat strutured fats from) the web (Carlson et al., 2010). NELL began operation in January As of Marh 2011, NELL has built a knowledge base ontaining several million andidate beliefs whih it has extrated from the web with varying onfidene. Among these,
2 NELL has fairly high onfidene in approximately half a million, whih we refer to as NELL s (onfident) beliefs. NELL has lower onfidene in a few million others, whih we refer to as its andidate beliefs. NELL is given as input an ontology that defines hundreds of ategories (e.g., person, beverage, athlete, sport) and twoplae typed relations among these ategories (e.g., atheleteplayssport( athlete, sport )), whih it must learn to extrat from the web. It is also provided a set of 10 to 20 positive seed examples of eah suh ategory and relation, along with a downloaded olletion of 500 million web pages from the ClueWeb2009 orpus (Callan and Hoy, 2009) as unlabeled data, and aess to 100,000 queries eah day to Google s searh engine. Eah day, NELL has two tasks: (1) to extrat additional beliefs from the web to populate its growing knowledge base (KB) with instanes of the ategories and relations in its ontology, and (2) to learn to perform task 1 better today than it ould yesterday. We an measure its learning ompetene by allowing it to onsider the same text douments today as it did yesterday, and reording whether it extrats more beliefs, more aurately today than it did yesterday. NELL uses a largesale semisupervised multitask learning algorithm that ouples the training of over 1500 different lassifiers and extration methods (see (Carlson et al., 2010)). Although many of the details of NELL s learning method are not entral to this paper, two points should be noted. First, NELL is a multistrategy learning system, with omponents that learn from different views of the data (Blum and Mithell, 1998): for instane, one view uses orthographi features of a potential entity name (like ontains apitalized words ), and another uses freetext ontexts in whih the noun phrase is found (e.g., X frequently follows the bigram mayor of ). Seond, NELL is a bootstrapping system, whih selftrains on its growing olletion of onfident beliefs. 1.2 Knowledge Base Inferene: Horn Clauses Although NELL has now grown a sizable knowledge base, its ability to perform inferene over this knowledge base is urrently very limited. At present its only inferene method beyond simple inheritane HinesWard Eli Manning AthletePlays ForTeam AthletePlays ForTeam Steelers Giants TeamPlays InLeague TeamPlays InLeague TeamPlays InLeague Figure 1: An example subgraph. NFL MLB involves applying first order Horn lause rules to infer new beliefs from urrent beliefs. For example, it may use a Horn lause suh as AthletePlaysForTeam(x, y) (1) TeamPlaysInLeague(y, z) AthletePlaysInLeague(x,z) to infer that AthletePlaysInLeague(HinesWard,NFL), if it has already extrated the beliefs in the preonditions of the rule, with variables x, y and z bound to HinesWard, PittsburghSteelers and NFL respetively as shown in Figure 1. NELL urrently has a set of approximately 600 suh rules, whih it has learned by data mining its knowledge base of beliefs. Eah learned rule arries a onditional probability that its onlusion will hold, given that its preonditions are satisfied. NELL learns these Horn lause rules using a variant of the FOIL algorithm (Quinlan and CameronJones, 1993), heneforth NFOIL. NFOIL takes as input a set of positive and negative examples of a rule s onsequent (e.g., +AthletePlaysInLeague(HinesWard,NFL), AthletePlaysInLeague(HinesWard,NBA)), and uses a separateandonquer strategy to learn a set of Horn lauses that fit the data well. Eah Horn lause is learned by starting with a general rule and progressively speializing it, so that it still overs many positive examples but overs few negative examples. After a lause is learned, the examples overed by that lause are removed from the training set, and the proess repeats until no positive examples remain. Learning firstorder Horn lauses is omputationally expensive not only is the searh spae large, but some Horn lauses an be ostly to evaluate (Cohen and Page, 1995). NFOIL uses two triks to improve its salability. First, it assumes that the onsequent prediate is funtional e.g., that eah Athlete plays in at most one League. This means that expliit negative examples need not be
3 provided (Zelle et al., 1995): e.g., if AthletePlaysIn League(HinesWard,NFL) is a positive example, then AthletePlaysInLeague(HinesWard,z ) for any other value of z is negative. In general, this onstraint guides the searh algorithm toward Horn lauses that have fewer possible instantiations, and hene are less expensive to math. Seond, NFOIL uses relational pathfinding (Rihards and Mooney, 1992) to produe general rules i.e., the starting point for a lause of the form R(x, y) is found by looking at positive instanes R(a, b) of the onsequent, and finding a lause that orresponds to a boundedlength path of binary relations that link a to b. In the example above, a start lause might be the lause (1). As in FOIL, the lause is then (potentially) speialized by greedily adding additional onditions (like ProfessionalAthlete(x)) or by replaing variables with onstants (eg, replaing z with NFL). For eah NFOIL rule, an estimated onditional probability ˆP (onlusion preonditions) is alulated using a Dirihlet prior aording to ˆP = (N + + m prior)/(n + + N + m) (2) where N + is the number of positive instanes mathed by this rule in the FOIL training data, N is the number of negative instanes mathed, m = 5 and prior = 0.5. As the results below show, NFOIL generally learns a small number of highpreision inferene rules. One important role of these inferene rules is that they ontribute to the bootstrapping proedure, as inferenes made by NFOIL inrease either the number of andidate beliefs, or (if the inferene is already a andidate) improve NELL s onfidene in andidate beliefs. 1.3 Knowledge Base Inferene: Graph Random Walks In this paper, we onsider an alternative approah, based on the Path Ranking Algorithm (PRA) of Lao and Cohen (2010b), desribed in detail below. PRA learns to rank graph nodes y relative to a query node x. PRA begins by enumerating a large set of boundedlength edgelabeled path types, similar to the initial lauses used in NELL s variant of FOIL. These path types are treated as ranking experts, eah performing a random walk through the graph, onstrained to follow that sequene of edge types, and ranking nodes y by their weights in the resulting distribution. Finally PRA ombines these experts using logisti regression. As an example, onsider a path from x to y via the sequene of edge types isa, isa 1 (the inverse of isa), and AthletePlaysInLeague, whih orresponds to the Horn lause isa(x, ) isa 1 (, x ) (3) AthletePlaysInLeague(x, y) AthletePlaysInLeague(x, y) Suppose a random walk starts at a query node x (say x=hinesward). If HinesWard is linked to the single onept node ProfessionalAthlete via isa, the walk will reah that node with probability 1 after one step. If A is the set of ProfessionalAthlete s in the KB, then after two steps, the walk will have probability 1/ A of being at any x A. If L is the set of athleti leagues and l L, let A l be the set of athletes in league l: after three steps, the walk will have probability A l / A of being at any point y L. In short, the ranking assoiated with this path gives the prior probability of a value y being an athleti league for x whih is useful as a feature in a ombined ranking method, although not by itself a highpreision inferene rule. Note that the rankings produed by this expert will hange as the knowledge base evolves for instane, if the system learns about proportionally more soer players than hokey players over time, then the league rankings for the path of lause (3) will hange. Also, the ranking is speifi to the query node x. For instane, suppose the KB ontains fats whih reflet the ambiguity of the team name Giants 1 as in Figure 1. Then the path for lause (1) above will give lower weight to y = NFL for x = EliManning than to y = NFL for x = HinesWard. The main ontribution of this paper is to introdue and evaluate PRA as an algorithm for making probabilisti inferene in large KBs. Compared to Horn lause inferene, the key harateristis of this new inferene method are as follows: The evidene in support of inferring a relation instane R(a, b) is based on many existing paths between a and b in the urrent KB, ombined using a learned logisti funtion. 1 San Franiso s MajorLeague Baseball and New York s National Football League teams are both alled the Giants.
4 The onfidene in an inferene is sensitive to the urrent state of the knowledge base, and the speifi entities being queried (sine the paths used in the inferene have these properties). Experimentally, the inferene method yields many more moderatelyonfident inferenes than the Horn lauses learned by NFOIL. The learning and inferene are more effiient than NFOIL, in part beause we an exploit effiient approximation shemes for random walks (Lao and Cohen, 2010a). The resulting inferene is as fast as 10 milliseonds per query on average. The Path Ranking Algorithm (PRA) we use is similar to that desribed elsewhere (Lao and Cohen, 2010b), exept that to ahieve effiient model learning, the paths between a and b are determined by the statistis from a population of training queries rather than enumerated ompletely. PRA uses random walks to generate relational features on graph data, and ombine them with a logisti regression model. Compared to other relational models (e.g. FOIL, Markov Logi Networks), PRA is extremely effiient at link predition or retrieval tasks, in whih we are interested in identifying top links from a large number of andidates, instead of fousing on a partiular node pair or joint inferenes. 1.4 Related Work The TextRunner system (Cafarella et al., 2006) answers list queries on a large knowledge base produed by open domain information extration. Spreading ativation is used to measure the loseness of any node to the query term nodes. This approah is similar to the random walk with restart approah whih is used as a baseline in our experiment. The FatRank system (Jain and Pantel, 2010) ompares different ways of onstruting random walks, and ombining them with extration sores. However, the shortoming of both approahes is that they ignore edge type information, whih is important for ahieving high auray preditions. The HOLMES system (Shoenmakers et al., 2008) derives new assertions using a few manually written inferene rules. A Markov network orresponding to the grounding of these rules to the knowledge base is onstruted for eah query, and then belief propagation is used for inferene. In omparison, our proposed approah disovers inferene rules automatially from training data. Below we desribe our approah in greater detail, provide experimental evidene of its value for performing inferene in NELL s knowledge base, and disuss impliations of this work and diretions for future researh. 2 Approah In this setion, we first desribe how we formulate link (relation) predition on a knowledge base as a ranking task. Then we review the Path Ranking Algorithm (PRA) introdued by Lao and Cohen (2010b; 2010a). After that, we desribe two improvement to the PRA method so that it is more suitable for the task of link predition on knowledge base. The first improvement helps PRA deal with the large number of relations whih is typial of large knowledge bases. The seond improvement aims at improving the quality of inferene by applying low variane sampling. 2.1 Learning with NELL s Knowledge Base For eah relation R in the knowledge base we train a model for the link predition task: given a onept x, find all other onepts y whih potentially have the relation R(x, y). This predition is made based on an existing knowledge base extrated imperfetly from the web. Although a model an potentially benefit from prediting multiple relations jointly, suh joint inferene is beyond the sope of this work. To ensure reasonable number of training instanes, we generate labeled training queries from 48 relations whih have more than 100 instanes in the knowledge base. We reate two tasks for eah relation i.e., prediting y given x and prediting x given y yielding 96 tasks in all. Eah node x whih has relation R in the knowledge base with any other node is treated as a training query, the atual nodes y in the knowledge base known to satisfy R(x, y) are treated as labeled positive examples, and any other nodes are treated as negative examples.
5 2.2 Path Ranking Algorithm Review We now review the Path Ranking Algorithm introdued by Lao and Cohen (2010b). A relation path P is defined as a sequene of relations R 1... R l, and in order to emphasize the types assoiated with eah step, P an also be written as R T 1 R 0... l Tl, where T i = range(r i ) = domain(r i+1 ), and we also define domain(p ) T 0, range(p ) T l. In the experiments in this paper, there is only one type of nodes alled onept, whih an be onneted through different types of relations. In this notation, relations like the team ertain player plays for, and the league ertain player s team is in an be expressed by the paths below (respetively): P 1 : onept AtheletePlayesForTeam onept P 2 : onept AtheletePlayesForTeam onept TeamPlaysInLeagure onept For any relation path P = R 1... R l and a seed node s domain(p ), a path onstrained random walk defines a distribution h s,p reursively as follows. If P is the empty path, then define { 1, if e = s h s,p (e) = (4) 0, otherwise If P = R 1... R l is nonempty, then let P = R 1... R l 1, and define h s,p (e) = h s,p (e ) P (e e ; R l ), (5) e range(p ) where P (e e ; R l ) = R l(e,e) R l (e, ) is the probability of reahing node e from node e with a one step random walk with edge type R l. R(e, e) indiates whether there exists an edge with type R that onnet e to e. More generally, given a set of paths P 1,..., P n, one ould treat eah h s,pi (e) as a path feature for the node e, and rank nodes by a linear model θ 1 h s,p1 (e) + θ 2 h s,p2 (e) +... θ n h s,pn (e) where θ i are appropriate weights for the paths. This gives a ranking of nodes e related to the query node s by the following soring funtion sore(e; s) = P P l h s,p (e)θ P, (6) where P l is the set of relation paths with length l. Given a relation R and a set of node pairs {(s i, t i )}, we an onstrut a training dataset D = {(x i, r i )}, where x i is a vetor of all the path features for the pair (s i, t i ) i.e., the jth omponent of x i is h si,p j (t i ), and r i indiates whether R(s i, t i ) is true. Parameter θ is estimated by maximizing the following regularized objetive funtion O(θ) = i o i (θ) λ 1 θ 1 λ 2 θ 2, (7) where λ 1 ontrols L 1 regularization to help struture seletion, and λ 2 ontrols L 2 regularization to prevent overfitting. o i (θ) is a perinstane objetive funtion, whih is defined as o i (θ) = w i [r i ln p i + (1 y i ) ln(1 p i )], (8) where p i is the predited relevane defined as p i = p(r i = 1 x i ; θ) = exp(θt x i ) 1+exp(θ T x i ), and w i is an importane weight to eah example. A biased sampling proedure selets only a small subset of negative samples to be inluded in the objetive (see (Lao and Cohen, 2010b) for detail). 2.3 DataDriven Path Finding In prior work with PRA, P l was defined as all relation paths of length at most l. When the number of edge types is small, one an generate P l by enumeration; however, for domains with a large number of relations (e.g., a knowledge base), it is impratial to enumerate all possible relation paths even for small l. For instane, if the number of relations related to eah node is 100, even the number of length three paths easily reahes millions. For other domains like parsed natural language sentenes, useful paths an be as long as ten relations (Minkov and Cohen, 2008). In this ase, even with smaller number of possible relations, the total number of relation path is still too large for systemati enumeration. In order to apply PRA to these domains, we modify the path generation proedure in PRA to produe only paths whih are potentially useful for the task. Define a query s to be supporting a path P if h s,p (e) 0 for any entity e. We require that any node reated during path finding needs to
6 Table 1: Number of paths in PRA models of maximum path length 3 and 4. Averaged over 96 tasks. l=3 l=4 all paths up to length L 15, 376 1, 906, 624 +query support α = ever reah a target entity L 1 regularization be supported by at least a fration α of the training queries s i, as well as being of length no more than l (In the experiments, we set α = 0.01) We also require that a path inluded in the PRA model only if it retrieves at least one target entity t i in the training set. As we an see from Table 1, together these two onstraints dramatially redue the number of paths that need to be onsidered, relative to systematially enumerating all possible paths. L1 regularization redues the size of the model even more. The idea of finding paths that onnets nodes in a graph is not new. It has been embodied previously in firstorder learning systems (Rihards and Mooney, 1992) as well as NFOIL, and relational database searhing systems (Bhalotia et al., 2002). These approahes onsider a single query during path finding. In omparison, the datadriven path finding method we desribed here uses statistis from a population of queries, and therefore an potentially determine the importane of a path more reliably. 2.4 LowVariane Sampling Lao and Cohen (2010a) previously showed that sampling tehniques like finger printing and partile filtering an signifiantly speedup random walk without sarifiing retrieval quality. However, the sampling proedures an indue a loss of diversity in the partile population. For example, onsider a node in the graph with just two out links with equal weights, and suppose we are required to generate two walkers starting from this node. A disappointing result is that with 50 perent hane both walkers will follow the same branh, and leave the other branh with no probability mass. To overome this problem, we apply a tehnique alled LowVariane Sampling (LVS) (Thrun et al., 2005), whih is ommonly used in robotis to improve the quality of sampling. Instead of generating independent samples from a distribution, LVS uses a single random number to generate all Table 2: Comparing PRA with RWR models. MRRs and training times are averaged over 96 tasks. l=2 l=3 MRR Time MRR Time RWR(no train) RWR s s PRA s s samples, whih are evenly distributed aross the whole distribution. Note that given a distribution P (x), any number r in [0, 1] points to exatly one x value, namely x = arg min j m=1..j P (m) r. Suppose we want to generate M samples from P (x). LVS first generates a random number r in the interval [0, M 1 ]. Then LVS repeatedly adds the fixed amount M 1 to r and hooses x values orresponding to the resulting numbers. 3 Results This setion reports empirial results of applying random walk inferene to NELL s knowledge base after the 165th iteration of its learning proess. We first investigate PRA s behavior by ross validation on the training queries. Then we ompare PRA and NFOIL s ability to reliably infer new beliefs, by leveraging the Amazon Mehanial Turk servie. 3.1 Cross Validation on the Training Queries Random Walk with Restart (RWR) (also alled personalized PageRank (Haveliwala, 2002)) is a generalpurpose graph proximity measure whih has been shown to be fairly suessful for many types of tasks. We ompare PRA to two versions of RWR on the 96 tasks of link predition with NELL s knowledge base. The two baseline methods are an untrained RWR model and a trained RWR model as desribed by Lao and Cohen (2010b). (In brief, in the trained RWR model, the walker will probabilistially prefer to follow edges assoiated with different labels, where the weight for eah edge label is hosen to minimize a loss funtion, suh as Equation 7. In the untrained model, edge weights are uniform.) We explored a range of values for the regularization parameters L 1 and L 2 using ross validation on the training data, and we fix both L 1 and L 2 parameters to for all tasks. The
7 MRR k 10k Exat 10k Independent Fingerprinting Low Variane Fingerprinting Independent Filtering Low Variane Filtering Random Walk Speedup Figure 2: Compare inferene speed and quality over 96 tasks. The speedup is relative to exat inferene, whih is on average 23ms per query. 1k 100 maximum path length is fixed to 3. 2 Table 2 ompares the three methods using 5fold ross validation and the Mean Reiproal Rank (MRR) 3 measure, whih is defined as the inverse rank of the highest ranked relevant result in a set of results. If the the first returned result is relevant, then MRR is 1.0, otherwise, it is smaller than 1.0. Supervised training an signifiantly improve retrieval quality (pvalue= omparing untrained and trained RWR), and leveraging path information an produe further improvement (pvalue= omparing trained RWR with PRA). The average training time for a prediate is only a few seonds. We also investigate the effet of lowvariane sampling on the quality of predition. Figure 2 ompares independent and low variane sampling when applied to finger printing and partile filtering (Lao and Cohen, 2010a). The horizontal axis orresponds to the speedup of random walk ompared with exat inferene, and the vertial axis measures the quality of predition by MRR with three fold ross validation on the training query set. Lowvariane sampling an improve predition for both finger 2 Results with maximum length 4 are not reported here. Generally models with length 4 paths produe slightly better results, but are 45 times slower to train 3 For a set of queries Q, MRR = 1 Q q Q 1k 1 rank of the first orret answer for q printing and partile filtering. The numbers on the urves indiate the number of partiles (or walkers). When using a large number of partiles, the partile filtering methods onverge to the exat inferene. Interestingly, when using a large number of walkers, the finger printing methods produe even better predition quality than exat inferene. Lao and Cohen notied a similar improvement on retrieval tasks, and onjetured that it is beause the sampling inferene imposes a regularization penalty on longer relation paths (2010a). 3.2 Evaluation by Mehanial Turk The rossvalidation result above assumes that the knowledge base is omplete and orret, whih we know to be untrue. To aurately ompare PRA and NFOIL s ability to reliably infer new beliefs from an imperfet knowledge base, we use human assessments obtained from Amazon Mehanial Turk. To limit labeling osts, and sine our goal is to improve the performane of NELL, we do not inlude RWRbased approahes in this omparison. Among all the 24 funtional prediates, NFOIL disovers onfident rules for 8 of them (it produes no result for the other 16 prediates). Therefore, we ompare the quality of PRA to NFOIL on these 8 prediates only. Among all the 72 nonfuntional prediates whih NFOIL annot be applied to PRA exhibits a wide range of performane in rossvalidation. The are 43 tasks for whih PRA obtains MRR higher than 0.4 and builds a model with more than 10 path features. We randomly sampled 8 of these prediates to be evaluated by Amazon Mehanial Turk. Table 3 shows the top two weighted PRA features for eah task on whih NFOIL an suessfully learn rules. These PRA rules an be ategorized into broad overage rules whih behave like priors over orret answers (e.g. 12, 46, 15), aurate rules whih leverage speifi relation sequenes (e.g. 9, 11, 14), rules whih leverage information about the synonyms of the query node (e.g. 78, 10, 12), and rules whih leverage information from a loal neighborhood of the query node (e.g. 3, 1213, 16). The synonym paths are useful, beause an entity may have multiple names on the web. We find that all 17 general rules (no speialization) learned by NFOIL an be expressed as length two relation
8 Table 3: The top two weighted PRA paths for tasks on whih NFOIL disovers onfident rules. stands for onept. ID athleteplaysforteam leagueplayers PRA Path (Comment) athleteplaysforteam 1 (teams with many players in the athlete s league) 2 leagueteams teamagainstteam (teams that play against many teams in the athlete s league) 3 athleteplayssport players (the league that players of a ertain sport belong to) 4 isa isa 1 athleteplayssport 5 isa isa 1 6 stadiumloatedincity 7 stadiumhometeam 8 latitudelongitude teamhomestadium 9 teamplaysincity 10 teammember teamplaysincity 11 teamhomestadium (popular leagues with many players) athleteplayssport (popular sports of all the athletes) superpartoforganization teamhomestadium teamplayssport (popular sports of a ertain league) stadiumloatedincity (ity of the stadium with the same team) latitudelongitudeof stadiumloatedincity (ity of the stadium with the same loation) itystadiums (stadiums loated in the same ity with the query team) athleteplaysforteam stadiumloatedincity (ity of the team s home stadium) 12 teamhomestadium stadiumhometeam teamplaysinleague 13 teamplayssport players teamplaysagainstteam 14 teamplayssport 15 isa isa 1 16 teamplaysinleague teamplayssport teamhomestadium (home stadium of teams whih share players with the query) teamplaysincity (ity of teams with the same home stadium as the query) (the league that the query team s members belong to) teamplaysinleague (the league that the query team s ompeting team belongs to) (sports played by many teams) leagueteams teamplayssport (the sport played by other teams in the league) Table 4: Amazon Mehanial Turk evaluation for the promoted knowledge. Using paired ttest at task level, PRA is not statistially different than NFOIL for (pvalue=0.3), but is signifiantly better for (pvalue=0.003) PRA NFOIL Task P majority #Paths #Rules #Query athleteplaysforteam (+1) (+30) athleteplayssport (+30) stadiumloatedincity (+0) teamhomestadium (+0) teamplaysincity (+0) teamplaysinleague (+151) teamplayssport (+86) average teammember ompaniesheadquarteredin publiationjournalist produedby NFOIL does not produe results ompeteswith for nonfuntional prediates hasoffieincity teamwontrophy worksfor average
9 Table 5: Compare Mehanial Turk workers voted assessments with our gold standard labels based on 100 samples. AMT=F AMT=T Gold=F 25% 15% Gold=T 11% 49% paths suh as path 11. In omparison, PRA explores a feature spae with many length three paths. For eah relation R to be evaluated, we generate test queries s whih belong to domain(r). Queries whih appear in the training set are exluded. For eah query node s, we applied a trained model (either PRA or NFOIL) to generate a ranked list of andidate t nodes. For PRA, the andidates are sorted by their sores as in Eq. (6). For NFOIL, the andidates are sorted by the estimated auraies of the rules as in Eq. (2) (whih generate the andidates). Sine there are about 7 thousand (and 13 thousand) test queries s for eah funtional (and nonfuntional) prediate R, and there are (potentially) thousands of andidates t returned for eah query s, we annot evaluate all andidates of all queries. Therefore, we first sort the queries s for eah prediate R by the sores of their top ranked andidate t in desending order, and then alulate preisions at top 10, 100 and 1000 positions for the 1 ),..., where list of result R(s R,1, t R,1 1 ), R(s R,2, t R,2 s R,1 is the first query for prediate R, t R,1 1 is its first andidate, s R,2 is the seond query for prediate R, t R,2 1 is its first andidate, so on and so forth. To redue the labeling load, we judge all top 10 queries for eah prediate, but randomly sample 50 out of the top 100, and randomly sample 50 out of the top Eah belief is evaluated by 5 workers at Mehanial Turk, who are given assertions like Hines Ward plays for the team Steelers, as well as Google searh links for eah entity, and the ombination of both entities. Statistis shows that the workers spend on average 25 seonds to judge eah belief. We also remove some workers judgments whih are obviously inorret 4. We sampled 100 beliefs, and ompared their voted result to goldstandard labels produed by one author of this paper. Table 5 shows that 74% of the time the workers voted result agrees with our judgement. 4 Certain workers label all the questions with the same answer Table 4 shows the evaluation result. The P majority olumn shows for eah prediate the auray ahieved by the majority predition: given a query R(x,?), predit the y that most often satisfies R over all possible x in the knowledge base. Thus, the higher P majority is, the simpler the task. Prediting the funtional prediates is generally easier prediting the nonfuntional prediates. The #Query olumn shows the number of queries on whih NFOIL is able to math any of its rules, and hene produe a andidate belief. For most prediates, NFOIL is only able to produe results for at most a few hundred queries. In omparison, PRA is able to produe results for 6,599 queries on average for eah funtional prediate, and 12,519 queries on average for eah nonfuntional prediate. Although the preision at 10 of NFOIL is omparable to that of PRA, preision at 10 and at 1000 and are muh lower 5. The #Path olumn shows the number of paths learned by PRA, and the #Rule olumn shows the number of rules learned by NFOIL, with the numbers before brakets orrespond to unspeialized rules, and the numbers in brakets orrespond to speialized rules. Generally, speialized rules have muh smaller reall than unspeialized rules. Therefore, the PRA approah ahieves high reall partially by ombining a large number of unspeialized paths, whih orrespond to unspeialized rules. However, learning more aurate speialized paths is part of our future work. A signifiant advantage of PRA over NFOIL is that it an be applied to nonfuntional prediates. The last eight rows of Table 4 show PRA s performane on eight of these prediates. Compared to the result on funtional prediates, preisions at 10 and at 100 of nonfuntional prediates are slightly lower, but preisions at 1000 are omparable. We note that for some prediates preision at 1000 is better than at 100. After some investigation we found that for many relations, the top portion of the result list is more diverse: i.e. showing produts produed by different ompanies, journalist working at different publiations. 5 If a method makes k preditions, and k < n, then is the number orret out of the k preditions, divided by n
10 While the lower half of the result list is more homogeneous: i.e. showing relations onentrated on one or two ompanies/publiations. On the other hand, through the proess of labeling the Mehanial Turk workers seem to build up a prior about whih ompany/publiation is likely to have orret beliefs, and their judgments are positively biased towards these ompanies/publiations. These two fators ombined together result in positive bias towards the lower portion of the result list. In future work we hope to design a labeling strategy whih avoids this bias. 4 Conlusions and Future Work We have shown that a soft inferene proedure based on a ombination of onstrained, weighted, random walks through the knowledge base graph an be used to reliably infer new beliefs for the knowledge base. We applied this approah to a knowledge base of approximately 500,000 beliefs extrated imperfetly from the web by NELL. This new system improves signifiantly over NELL s earlier Hornlause learning and inferene method: it obtains nearly double the preision at rank 100. The inferene and learning are both very effiient our experiment shows that the inferene time is as fast as 10 milliseonds per query on average, and the training for a prediate takes only a few seonds. There are some prominent diretions for future work. First, inferene starting from both the query nodes and target nodes (Rihards and Mooney, 1992) an be muh more effiient in disovering long paths than just inferene from the query nodes. Seond, inferene starting from the target nodes of training queries is a potential way to disover speialized paths (with grounded nodes). Third, generalizing inferene paths to inferene trees or graphs an produe more expressive random walk inferene models. Overall, we believe that random walk is a promising way to sale up relational learning to domains with very large data sets. Aknowledgments This work was supported by NIH under grant R01GM081293, by NSF under grant IIS , by DARPA under awards FA and AF C0179, and by a gift from Google. We thank Geoffrey J. Gordon for the suggestion of applying low variane sampling to random walk inferene. We also thank Bryan Kisiel for help with the NELL system. Referenes Eugene Agihtein and Luis Gravano Snowball: extrating relations from large plaintext olletions. In Proeedings of the fifth ACM onferene on Digital libraries  DL 00, pages 85 94, New York, New York, USA. ACM Press. Mihele Banko, Mihael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni Open Information Extration from the Web. In IJCAI, pages Gaurav Bhalotia, Arvind Hulgeri, Charuta Nakhe, Soumen Chakrabarti, and S. Sudarshan Keyword searhing and browsing in databases using banks. ICDE, pages Avrim Blum and Tom Mithell Combining labeled and unlabeled data with otraining. In Proeedings of the eleventh annual onferene on Computational learning theory  COLT 98, pages , New York, New York, USA. ACM Press. MJ Cafarella, M Banko, and O Etzioni Relational Web Searh. In WWW. Jamie Callan and Mark Hoy Clueweb09 data set. Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hrushka Jr., and Tom M. Mithell Toward an Arhiteture for NeverEnding Language Learning. In AAAI. William W. Cohen and David Page Polynomial learnability and indutive logi programming: Methods and results. New Generation Comput., 13(3&4): Oren Etzioni, Mihael Cafarella, Doug Downey, Anamaria Popesu, Tal Shaked, Stephen Soderl, Daniel S. Weld, and Er Yates Unsupervised namedentity extration from the web: An experimental study. Artifiial Intelligene, 165: Taher H. Haveliwala Topisensitive pagerank. In WWW, pages Alpa Jain and Patrik Pantel Fatrank: Random walks on a web of fats. In COLING, pages Ni Lao and William W. Cohen. 2010a. Fast query exeution for retrieval models based on pathonstrained random walks. KDD. Ni Lao and William W. Cohen. 2010b. Relational retrieval using a ombination of pathonstrained random walks. Mahine Learning.
11 Einat Minkov and William W. Cohen Learning graph walk based similarity measures for parsed text. EMNLP, pages Patrik Pantel and Maro Pennahiotti Espresso: Leveraging Generi Patterns for Automatially Harvesting Semanti Relations. In ACL. J. Ross Quinlan and R. Mike CameronJones FOIL: A Midterm Report. In ECML, pages Bradley L. Rihards and Raymond J. Mooney Learning relations by pathfinding. In Proeedings of the Tenth National Conferene on Artifiial Intelligene (AAAI92), pages 50 55, San Jose, CA, July. Matthew Rihardson and Pedro Domingos Markov logi networks. Mahine Learning. Stefan Shoenmakers, Oren Etzioni, and Daniel S. Weld Saling Textual Inferene to the Web. In EMNLP, pages Rion Snow, Daniel Jurafsky, and Andrew Y. Ng Semanti Taxonomy Indution from Heterogenous Evidene. In ACL. Sebastian Thrun, Wolfram Burgard, and Dieter Fox Probabilisti Robotis (Intelligent Robotis and Autonomous Agents). The MIT Press. Alexander Yates, Mihele Banko, Matthew Broadhead, Mihael J. Cafarella, Oren Etzioni, and Stephen Soderland TextRunner: Open Information Extration on the Web. In HLTNAACL (Demonstrations), pages John M. Zelle, Cynthia A. Thompson, Mary Elaine Califf, and Raymond J. Mooney Induing logi programs without expliit negative examples. In Proeedings of the Fifth International Workshop on Indutive Logi Programming (ILP95), pages , Leuven, Belgium.
A Holistic Method for Selecting Web Services in Design of Composite Applications
A Holisti Method for Seleting Web Servies in Design of Composite Appliations Mārtiņš Bonders, Jānis Grabis Institute of Information Tehnology, Riga Tehnial University, 1 Kalu Street, Riga, LV 1658, Latvia,
More informationChannel Assignment Strategies for Cellular Phone Systems
Channel Assignment Strategies for Cellular Phone Systems Wei Liu Yiping Han Hang Yu Zhejiang University Hangzhou, P. R. China Contat: wliu5@ie.uhk.edu.hk 000 Mathematial Contest in Modeling (MCM) Meritorious
More informationHierarchical Clustering and Sampling Techniques for Network Monitoring
S. Sindhuja Hierarhial Clustering and Sampling Tehniques for etwork Monitoring S. Sindhuja ME ABSTRACT: etwork monitoring appliations are used to monitor network traffi flows. Clustering tehniques are
More informationA ContextAware Preference Database System
J. PERVASIVE COMPUT. & COMM. (), MARCH 005. TROUBADOR PUBLISHING LTD) A ContextAware Preferene Database System Kostas Stefanidis Department of Computer Siene, University of Ioannina,, kstef@s.uoi.gr Evaggelia
More informationOpen and Extensible Business Process Simulator
UNIVERSITY OF TARTU FACULTY OF MATHEMATICS AND COMPUTER SCIENCE Institute of Computer Siene Karl Blum Open and Extensible Business Proess Simulator Master Thesis (30 EAP) Supervisors: Luiano GaríaBañuelos,
More informationDiscovering Trends in Large Datasets Using Neural Networks
Disovering Trends in Large Datasets Using Neural Networks Khosrow Kaikhah, Ph.D. and Sandesh Doddameti Department of Computer Siene Texas State University San Maros, Texas 78666 Abstrat. A novel knowledge
More informationChapter 1 Microeconomics of Consumer Theory
Chapter 1 Miroeonomis of Consumer Theory The two broad ategories of deisionmakers in an eonomy are onsumers and firms. Eah individual in eah of these groups makes its deisions in order to ahieve some
More informationSebastián Bravo López
Transfinite Turing mahines Sebastián Bravo López 1 Introdution With the rise of omputers with high omputational power the idea of developing more powerful models of omputation has appeared. Suppose that
More information' R ATIONAL. :::~i:. :'.:::::: RETENTION ':: Compliance with the way you work PRODUCT BRIEF
' R :::i:. ATIONAL :'.:::::: RETENTION ':: Compliane with the way you work, PRODUCT BRIEF Inplae Management of Unstrutured Data The explosion of unstrutured data ombined with new laws and regulations
More informationWeighting Methods in Survey Sampling
Setion on Survey Researh Methods JSM 01 Weighting Methods in Survey Sampling Chiaohih Chang Ferry Butar Butar Abstrat It is said that a welldesigned survey an best prevent nonresponse. However, no matter
More informationRanking Community Answers by Modeling QuestionAnswer Relationships via Analogical Reasoning
Ranking Community Answers by Modeling QuestionAnswer Relationships via Analogial Reasoning XinJing Wang Mirosoft Researh Asia 4F Sigma, 49 Zhihun Road Beijing, P.R.China xjwang@mirosoft.om Xudong Tu,Dan
More informationBig Data Analysis and Reporting with Decision Tree Induction
Big Data Analysis and Reporting with Deision Tree Indution PETRA PERNER Institute of Computer Vision and Applied Computer Sienes, IBaI Postbox 30 11 14, 04251 Leipzig GERMANY pperner@ibaiinstitut.de,
More informationFindings and Recommendations
Contrating Methods and Administration Findings and Reommendations Finding 91 ESD did not utilize a formal written prequalifiations proess for seleting experiened design onsultants. ESD hose onsultants
More informationDeadlinebased Escalation in ProcessAware Information Systems
Deadlinebased Esalation in ProessAware Information Systems Wil M.P. van der Aalst 1,2, Mihael Rosemann 2, Marlon Dumas 2 1 Department of Tehnology Management Eindhoven University of Tehnology, The Netherlands
More informationUser s Guide VISFIT: a computer tool for the measurement of intrinsic viscosities
File:UserVisfit_2.do User s Guide VISFIT: a omputer tool for the measurement of intrinsi visosities Version 2.a, September 2003 From: Multiple Linear LeastSquares Fits with a Common Interept: Determination
More informationImproved Vehicle Classification in Long Traffic Video by Cooperating Tracker and Classifier Modules
Improved Vehile Classifiation in Long Traffi Video by Cooperating Traker and Classifier Modules Brendan Morris and Mohan Trivedi University of California, San Diego San Diego, CA 92093 {b1morris, trivedi}@usd.edu
More informationBUILDING A SPAM FILTER USING NAÏVE BAYES. CIS 391 Intro to AI 1
BUILDING A SPAM FILTER USING NAÏVE BAYES 1 Spam or not Spam: that is the question. From: "" Subjet: real estate is the only way... gem oalvgkay Anyone an buy real estate with no
More informationRecommending Questions Using the MDLbased Tree Cut Model
WWW 2008 / Refereed Trak: Data Mining  Learning April 225, 2008 Beijing, China Reommending Questions Using the MDLbased Tree Cut Model Yunbo Cao,2, Huizhong Duan, ChinYew Lin 2, Yong Yu, and HsiaoWuen
More informationGranular Problem Solving and Software Engineering
Granular Problem Solving and Software Engineering Haibin Zhu, Senior Member, IEEE Department of Computer Siene and Mathematis, Nipissing University, 100 College Drive, North Bay, Ontario, P1B 8L7, Canada
More informationStatic Fairness Criteria in Telecommunications
Teknillinen Korkeakoulu ERIKOISTYÖ Teknillisen fysiikan koulutusohjelma 92002 Mat208 Sovelletun matematiikan erikoistyöt Stati Fairness Criteria in Teleommuniations Vesa Timonen, email: vesatimonen@hutfi
More informationBehavior AnalysisBased Learning Framework for Host Level Intrusion Detection
Behavior AnalysisBased Learning Framework for Host Level Intrusion Detetion Haiyan Qiao, Jianfeng Peng, Chuan Feng, Jerzy W. Rozenblit Eletrial and Computer Engineering Department University of Arizona
More informationInterpretable Fuzzy Modeling using MultiObjective Immune Inspired Optimization Algorithms
Interpretable Fuzzy Modeling using MultiObjetive Immune Inspired Optimization Algorithms Jun Chen, Mahdi Mahfouf Abstrat In this paper, an immune inspired multiobjetive fuzzy modeling (IMOFM) mehanism
More informationTHE PERFORMANCE OF TRANSIT TIME FLOWMETERS IN HEATED GAS MIXTURES
Proeedings of FEDSM 98 998 ASME Fluids Engineering Division Summer Meeting June 225, 998 Washington DC FEDSM98529 THE PERFORMANCE OF TRANSIT TIME FLOWMETERS IN HEATED GAS MIXTURES John D. Wright Proess
More information) ( )( ) ( ) ( )( ) ( ) ( ) (1)
OPEN CHANNEL FLOW Open hannel flow is haraterized by a surfae in ontat with a gas phase, allowing the fluid to take on shapes and undergo behavior that is impossible in a pipe or other filled onduit. Examples
More informationFIRE DETECTION USING AUTONOMOUS AERIAL VEHICLES WITH INFRARED AND VISUAL CAMERAS. J. Ramiro Martínezde Dios, Luis Merino and Aníbal Ollero
FE DETECTION USING AUTONOMOUS AERIAL VEHICLES WITH INFRARED AND VISUAL CAMERAS. J. Ramiro Martínezde Dios, Luis Merino and Aníbal Ollero Robotis, Computer Vision and Intelligent Control Group. University
More informationImproved SOMBased HighDimensional Data Visualization Algorithm
Computer and Information Siene; Vol. 5, No. 4; 2012 ISSN 19138989 EISSN 19138997 Published by Canadian Center of Siene and Eduation Improved SOMBased HighDimensional Data Visualization Algorithm Wang
More informationRATING SCALES FOR NEUROLOGISTS
RATING SCALES FOR NEUROLOGISTS J Hobart iv22 WHY Correspondene to: Dr Jeremy Hobart, Department of Clinial Neurosienes, Peninsula Medial Shool, Derriford Hospital, Plymouth PL6 8DH, UK; Jeremy.Hobart@
More informationA Comparison of Service Quality between Private and Public Hospitals in Thailand
International Journal of Business and Soial Siene Vol. 4 No. 11; September 2013 A Comparison of Servie Quality between Private and Hospitals in Thailand Khanhitpol Yousapronpaiboon, D.B.A. Assistant Professor
More informationNeural networkbased Load Balancing and Reactive Power Control by Static VAR Compensator
nternational Journal of Computer and Eletrial Engineering, Vol. 1, No. 1, April 2009 Neural networkbased Load Balaning and Reative Power Control by Stati VAR Compensator smail K. Said and Marouf Pirouti
More informationIntelligent Measurement Processes in 3D Optical Metrology: Producing More Accurate Point Clouds
Intelligent Measurement Proesses in 3D Optial Metrology: Produing More Aurate Point Clouds Charles Mony, Ph.D. 1 President Creaform in. mony@reaform3d.om Daniel Brown, Eng. 1 Produt Manager Creaform in.
More informationRecovering Articulated Motion with a Hierarchical Factorization Method
Reovering Artiulated Motion with a Hierarhial Fatorization Method Hanning Zhou and Thomas S Huang University of Illinois at UrbanaChampaign, 405 North Mathews Avenue, Urbana, IL 680, USA {hzhou, huang}@ifpuiuedu
More informationExploiting Relative Addressing and Virtual Overlays in Ad Hoc Networks with Bandwidth and Processing Constraints
Exploiting Relative Addressing and Virtual Overlays in Ad Ho Networks with Bandwidth and Proessing Constraints Maro Aurélio Spohn Computer Siene Department University of California at Santa Cruz Santa
More informationchapter > Make the Connection Factoring CHAPTER 4 OUTLINE Chapter 4 :: Pretest 374
CHAPTER hapter 4 > Make the Connetion 4 INTRODUCTION Developing seret odes is big business beause of the widespread use of omputers and the Internet. Corporations all over the world sell enryption systems
More informationOpenScape 4000 CSTA V7 Connectivity Adapter  CSTA III, Part 2, Version 4.1. Developer s Guide A31003G9310I200176D1
OpenSape 4000 CSTA V7 Connetivity Adapter  CSTA III, Part 2, Version 4.1 Developer s Guide A31003G9310I200176 Our Quality and Environmental Management Systems are implemented aording to the requirements
More informationPattern Recognition Techniques in Microarray Data Analysis
Pattern Reognition Tehniques in Miroarray Data Analysis Miao Li, Biao Wang, Zohreh Momeni, and Faramarz Valafar Department of Computer Siene San Diego State University San Diego, California, USA faramarz@sienes.sdsu.edu
More informationAn Enhanced Critical Path Method for Multiple Resource Constraints
An Enhaned Critial Path Method for Multiple Resoure Constraints ChangPin Lin, HungLin Tai, and ShihYan Hu Abstrat Traditional Critial Path Method onsiders only logial dependenies between related ativities
More informationCapacity at Unsignalized TwoStage Priority Intersections
Capaity at Unsignalized TwoStage Priority Intersetions by Werner Brilon and Ning Wu Abstrat The subjet of this paper is the apaity of minorstreet traffi movements aross major divided fourlane roadways
More informationA Keyword Filters Method for Spam via Maximum Independent Sets
Vol. 7, No. 3, May, 213 A Keyword Filters Method for Spam via Maximum Independent Sets HaiLong Wang 1, FanJun Meng 1, HaiPeng Jia 2, JinHong Cheng 3 and Jiong Xie 3 1 Inner Mongolia Normal University 2
More informationAn Efficient Network Traffic Classification Based on Unknown and Anomaly Flow Detection Mechanism
An Effiient Network Traffi Classifiation Based on Unknown and Anomaly Flow Detetion Mehanism G.Suganya.M.s.,B.Ed 1 1 Mphil.Sholar, Department of Computer Siene, KG College of Arts and Siene,Coimbatore.
More informationA Design Environment for Migrating Relational to Object Oriented Database Systems
To appear in: 1996 International Conferene on Software Maintenane (ICSM 96); IEEE Computer Soiety, 1996 A Design Environment for Migrating Relational to Objet Oriented Database Systems Jens Jahnke, Wilhelm
More informationWORKFLOW CONTROLFLOW PATTERNS A Revised View
WORKFLOW CONTROLFLOW PATTERNS A Revised View Nik Russell 1, Arthur H.M. ter Hofstede 1, 1 BPM Group, Queensland University of Tehnology GPO Box 2434, Brisbane QLD 4001, Australia {n.russell,a.terhofstede}@qut.edu.au
More informationScalable Hierarchical Multitask Learning Algorithms for Conversion Optimization in Display Advertising
Salable Hierarhial Multitask Learning Algorithms for Conversion Optimization in Display Advertising Amr Ahmed Google amra@google.om Abhimanyu Das Mirosoft Researh abhidas@mirosoft.om Alexander J. Smola
More informationi_~f e 1 then e 2 else e 3
A PROCEDURE MECHANISM FOR BACKTRACK PROGRAMMING* David R. HANSON + Department o Computer Siene, The University of Arizona Tuson, Arizona 85721 One of the diffiulties in using nondeterministi algorithms
More informationProcurement auctions are sometimes plagued with a chosen supplier s failing to accomplish a project successfully.
Deision Analysis Vol. 7, No. 1, Marh 2010, pp. 23 39 issn 15458490 eissn 15458504 10 0701 0023 informs doi 10.1287/dea.1090.0155 2010 INFORMS Managing Projet Failure Risk Through Contingent Contrats
More informationRobust Classification and Tracking of Vehicles in Traffic Video Streams
Proeedings of the IEEE ITSC 2006 2006 IEEE Intelligent Transportation Systems Conferene Toronto, Canada, September 1720, 2006 TC1.4 Robust Classifiation and Traking of Vehiles in Traffi Video Streams
More informationChapter 5 Single Phase Systems
Chapter 5 Single Phase Systems Chemial engineering alulations rely heavily on the availability of physial properties of materials. There are three ommon methods used to find these properties. These inlude
More informationFrom a strategic view to an engineering view in a digital enterprise
Digital Enterprise Design & Management 2013 February 1112, 2013 Paris From a strategi view to an engineering view in a digital enterprise The ase of a multiountry Telo Hervé Paault Orange Abstrat In
More informationGABOR AND WEBER LOCAL DESCRIPTORS PERFORMANCE IN MULTISPECTRAL EARTH OBSERVATION IMAGE DATA ANALYSIS
HENRI COANDA AIR FORCE ACADEMY ROMANIA INTERNATIONAL CONFERENCE of SCIENTIFIC PAPER AFASES 015 Brasov, 830 May 015 GENERAL M.R. STEFANIK ARMED FORCES ACADEMY SLOVAK REPUBLIC GABOR AND WEBER LOCAL DESCRIPTORS
More informationClassical Electromagnetic Doppler Effect Redefined. Copyright 2014 Joseph A. Rybczyk
Classial Eletromagneti Doppler Effet Redefined Copyright 04 Joseph A. Rybzyk Abstrat The lassial Doppler Effet formula for eletromagneti waves is redefined to agree with the fundamental sientifi priniples
More informationAn integrated optimization model of a Closed Loop Supply Chain under uncertainty
ISSN 18166075 (Print), 18180523 (Online) Journal of System and Management Sienes Vol. 2 (2012) No. 3, pp. 917 An integrated optimization model of a Closed Loop Supply Chain under unertainty Xiaoxia
More informationprotection p1ann1ng report
f1re~~ protetion p1ann1ng report BUILDING CONSTRUCTION INFORMATION FROM THE CONCRETE AND MASONRY INDUSTRIES Signifiane of Fire Ratings for Building Constrution NO. 3 OF A SERIES The use of fireresistive
More informationAgile ALM White Paper: Redefining ALM with Five Key Practices
Agile ALM White Paper: Redefining ALM with Five Key Praties by Ethan Teng, Cyndi Mithell and Chad Wathington 2011 ThoughtWorks ln. All rights reserved www.studios.thoughtworks.om Introdution The pervasiveness
More informationOutline. Planning. Search vs. Planning. Search vs. Planning Cont d. Search vs. planning. STRIPS operators Partialorder planning.
Outline Searh vs. planning Planning STRIPS operators Partialorder planning Chapter 11 Artifiial Intelligene, lp4 2005/06, Reiner Hähnle, partly based on AIMA Slides Stuart Russell and Peter Norvig, 1998
More informationChapter 6 A N ovel Solution Of Linear Congruenes Proeedings NCUR IX. (1995), Vol. II, pp. 708{712 Jerey F. Gold Department of Mathematis, Department of Physis University of Utah Salt Lake City, Utah 84112
More informationFOOD FOR THOUGHT Topical Insights from our Subject Matter Experts
FOOD FOR THOUGHT Topial Insights from our Sujet Matter Experts DEGREE OF DIFFERENCE TESTING: AN ALTERNATIVE TO TRADITIONAL APPROACHES The NFL White Paper Series Volume 14, June 2014 Overview Differene
More informationAvailability, Reliability, Maintainability, and Capability
Availability, Reliability, Maintainability, and Capability H. Paul Barringer, P.E. Barringer & Assoiates, In. Humble, TX Triplex Chapter Of The Vibrations Institute Hilton Hotel Beaumont, Texas February
More informationIn order to be able to design beams, we need both moments and shears. 1. Moment a) From direct design method or equivalent frame method
BEAM DESIGN In order to be able to design beams, we need both moments and shears. 1. Moment a) From diret design method or equivalent frame method b) From loads applied diretly to beams inluding beam weight
More informationCustomer Efficiency, Channel Usage and Firm Performance in Retail Banking
Customer Effiieny, Channel Usage and Firm Performane in Retail Banking Mei Xue Operations and Strategi Management Department The Wallae E. Carroll Shool of Management Boston College 350 Fulton Hall, 140
More informationChemical Equilibrium. Chemical Equilibrium. Chemical Equilibrium. Chemical Equilibriu m. Chapter 14
Chapter 14 Chemial Equilibrium Chemial Equilibriu m Muh like water in a Ushaped tube, there is onstant mixing bak and forth through the lower portion of the tube. reatants produts It s as if the forward
More informationarxiv:astroph/0304006v2 10 Jun 2003 Theory Group, MS 50A5101 Lawrence Berkeley National Laboratory One Cyclotron Road Berkeley, CA 94720 USA
LBNL52402 Marh 2003 On the Speed of Gravity and the v/ Corretions to the Shapiro Time Delay Stuart Samuel 1 arxiv:astroph/0304006v2 10 Jun 2003 Theory Group, MS 50A5101 Lawrene Berkeley National Laboratory
More informationSoftware Ecosystems: From Software Product Management to Software Platform Management
Software Eosystems: From Software Produt Management to Software Platform Management Slinger Jansen, Stef Peeters, and Sjaak Brinkkemper Department of Information and Computing Sienes Utreht University,
More information3 Game Theory: Basic Concepts
3 Game Theory: Basi Conepts Eah disipline of the soial sienes rules omfortably ithin its on hosen domain: : : so long as it stays largely oblivious of the others. Edard O. Wilson (1998):191 3.1 and and
More informationChapter 1: Introduction
Chapter 1: Introdution 1.1 Pratial olumn base details in steel strutures 1.1.1 Pratial olumn base details Every struture must transfer vertial and lateral loads to the supports. In some ases, beams or
More informationA ThreeHybrid Treatment Method of the Compressor's Characteristic Line in Performance Prediction of Power Systems
A ThreeHybrid Treatment Method of the Compressor's Charateristi Line in Performane Predition of Power Systems A ThreeHybrid Treatment Method of the Compressor's Charateristi Line in Performane Predition
More informationSymmetric Subgame Perfect Equilibria in Resource Allocation
Proeedings of the TwentySixth AAAI Conferene on Artifiial Intelligene Symmetri Subgame Perfet Equilibria in Resoure Alloation Ludek Cigler Eole Polytehnique Fédérale de Lausanne Artifiial Intelligene
More informationImpact Simulation of Extreme Wind Generated Missiles on Radioactive Waste Storage Facilities
Impat Simulation of Extreme Wind Generated issiles on Radioative Waste Storage Failities G. Barbella Sogin S.p.A. Via Torino 6 00184 Rome (Italy), barbella@sogin.it Abstrat: The strutural design of temporary
More informationThe Optimal Deterrence of Tax Evasion: The Tradeoff Between Information Reporting and Audits
The Optimal Deterrene of Tax Evasion: The Tradeoff Between Information Reporting and Audits Yulia Paramonova Department of Eonomis, University of Mihigan Otober 30, 2014 Abstrat Despite the widespread
More informationA Survey of Usability Evaluation in Virtual Environments: Classi cation and Comparison of Methods
Doug A. Bowman bowman@vt.edu Department of Computer Siene Virginia Teh Joseph L. Gabbard Deborah Hix [ jgabbard, hix]@vt.edu Systems Researh Center Virginia Teh A Survey of Usability Evaluation in Virtual
More informationMagnetic Materials and Magnetic Circuit Analysis
Chapter 7. Magneti Materials and Magneti Ciruit Analysis Topis to over: 1) Core Losses 2) Ciruit Model of Magneti Cores 3) A Simple Magneti Ciruit 4) Magneti Ciruital Laws 5) Ciruit Model of Permanent
More informationComputational Analysis of Two Arrangements of a Central GroundSource Heat Pump System for Residential Buildings
Computational Analysis of Two Arrangements of a Central GroundSoure Heat Pump System for Residential Buildings Abstrat Ehab Foda, Ala Hasan, Kai Sirén Helsinki University of Tehnology, HVAC Tehnology,
More informationElectrician'sMathand BasicElectricalFormulas
Eletriian'sMathand BasiEletrialFormulas MikeHoltEnterprises,In. 1.888.NEC.CODE www.mikeholt.om Introdution Introdution This PDF is a free resoure from Mike Holt Enterprises, In. It s Unit 1 from the Eletrial
More informationAUDITING COST OVERRUN CLAIMS *
AUDITING COST OVERRUN CLAIMS * David PérezCastrillo # University of Copenhagen & Universitat Autònoma de Barelona Niolas Riedinger ENSAE, Paris Abstrat: We onsider a ostreimbursement or a ostsharing
More informationLearning Curves and Stochastic Models for Pricing and Provisioning Cloud Computing Services
T Learning Curves and Stohasti Models for Priing and Provisioning Cloud Computing Servies Amit Gera, Cathy H. Xia Dept. of Integrated Systems Engineering Ohio State University, Columbus, OH 4310 {gera.,
More informationTECHNOLOGYENHANCED LEARNING FOR MUSIC WITH IMAESTRO FRAMEWORK AND TOOLS
TECHNOLOGYENHANCED LEARNING FOR MUSIC WITH IMAESTRO FRAMEWORK AND TOOLS ICSRiM  University of Leeds Shool of Computing & Shool of Musi Leeds LS2 9JT, UK +441133432583 kia@imaestro.org www.imaestro.org,
More informationuniversity of illinois library AT URBANACHAMPAIGN BOOKSTACKS
university of illinois library AT URBANACHAMPAIGN BOOKSTACKS CENTRAL CIRCULATION BOOKSTACKS The person harging this material is responsible for its renewal or its return to the library from whih it was
More informationBayes Bluff: Opponent Modelling in Poker
Bayes Bluff: Opponent Modelling in Poker Finnegan Southey, Mihael Bowling, Brye Larson, Carmelo Piione, Neil Burh, Darse Billings, Chris Rayner Department of Computing Siene University of Alberta Edmonton,
More informationState of Maryland Participation Agreement for PreTax and Roth Retirement Savings Accounts
State of Maryland Partiipation Agreement for PreTax and Roth Retirement Savings Aounts DC4531 (08/2015) For help, please all 18009666355 www.marylandd.om 1 Things to Remember Complete all of the setions
More informationSupply chain coordination; A Game Theory approach
aepted for publiation in the journal "Engineering Appliations of Artifiial Intelligene" 2008 upply hain oordination; A Game Theory approah JeanClaude Hennet x and Yasemin Arda xx x LI CNRUMR 668 Université
More informationSLAbased Resource Allocation for Software as a Service Provider (SaaS) in Cloud Computing Environments
2 th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing SLAbased Resoure Alloation for Software as a Servie Provider (SaaS) in Cloud Computing Environments Linlin Wu, Saurabh Kumar
More informationSrinivas Bollapragada GE Global Research Center. Abstract
Sheduling Commerial Videotapes in Broadast Television Srinivas Bollapragada GE Global Researh Center Mihael Bussiek GAMS Development Corporation Suman Mallik University of Illinois at Urbana Champaign
More informationImpedance Method for Leak Detection in Zigzag Pipelines
10.478/v1004801000360 MEASUREMENT SCIENCE REVIEW, Volume 10, No. 6, 010 Impedane Method for Leak Detetion in igzag Pipelines A. LayEkuakille 1, P. Vergallo 1, A. Trotta 1 Dipartimento d Ingegneria
More information10.1 The Lorentz force law
Sott Hughes 10 Marh 2005 Massahusetts Institute of Tehnology Department of Physis 8.022 Spring 2004 Leture 10: Magneti fore; Magneti fields; Ampere s law 10.1 The Lorentz fore law Until now, we have been
More informationMATE: MPLS Adaptive Traffic Engineering
MATE: MPLS Adaptive Traffi Engineering Anwar Elwalid Cheng Jin Steven Low Indra Widjaja Bell Labs EECS Dept EE Dept Fujitsu Network Communiations Luent Tehnologies Univ. of Mihigan Calteh Pearl River,
More informationMarker Tracking and HMD Calibration for a Videobased Augmented Reality Conferencing System
Marker Traking and HMD Calibration for a Videobased Augmented Reality Conferening System Hirokazu Kato 1 and Mark Billinghurst 2 1 Faulty of Information Sienes, Hiroshima City University 2 Human Interfae
More informationRISKBASED IN SITU BIOREMEDIATION DESIGN JENNINGS BRYAN SMALLEY. A.B., Washington University, 1992 THESIS. Urbana, Illinois
RISKBASED IN SITU BIOREMEDIATION DESIGN BY JENNINGS BRYAN SMALLEY A.B., Washington University, 1992 THESIS Submitted in partial fulfillment of the requirements for the degree of Master of Siene in Environmental
More informationHierarchical Beta Processes and the Indian Buffet Process
Hierarhial Beta Proesses and the Indian Buffet Proess Romain Thibaux Dept. of EECS University of California, Berkeley Berkeley, CA 9472 Mihael I. Jordan Dept. of EECS and Dept. of Statistis University
More informationINCOME TAX WITHHOLDING GUIDE FOR EMPLOYERS
Virginia Department of Taxation INCOME TAX WITHHOLDING GUIDE FOR EMPLOYERS www.tax.virginia.gov 2614086 Rev. 07/14 * Table of Contents Introdution... 1 Important... 1 Where to Get Assistane... 1 Online
More informationpss Comparison of AlGaN/GaN MISHEMT powerbar designs solidi status physica
Phys. Status Solidi C 11, No. 3 4, 906 910 (2014) / DOI 10.1002/pss.201300490 Comparison of AlGaN/GaN MISHEMT powerbar designs physia pss www.pss.om urrent topis in solid state physis Steve Stoffels *,
More informationAsymmetric Error Correction and FlashMemory Rewriting using Polar Codes
1 Asymmetri Error Corretion and FlashMemory Rewriting using Polar Codes Eyal En Gad, Yue Li, Joerg Kliewer, Mihael Langberg, Anxiao (Andrew) Jiang and Jehoshua Bruk Abstrat We propose effiient oding shemes
More informationThe Application of Mamdani Fuzzy Model for Auto Zoom Function of a Digital Camera
(IJCSIS) International Journal of Computer Siene and Information Seurity, Vol. 6, No. 3, 2009 The Appliation of Mamdani Fuzzy Model for Auto Funtion of a Digital Camera * I. Elamvazuthi, P. Vasant Universiti
More informationA novel active mass damper for vibration control of bridges
IABMAS 08, International Conferene on Bridge Maintenane, Safety and Management, 37 July 008, Seoul, Korea A novel ative mass damper for vibration ontrol of bridges U. Starossek & J. Sheller Strutural
More informationprotection p1ann1ng report
( f1re protetion p1ann1ng report I BUILDING CONSTRUCTION INFORMATION FROM THE CONCRETE AND MASONRY INDUSTRIES NO. 15 OF A SERIES A Comparison of Insurane and Constrution Costs for LowRise Multifamily
More informationUse of Track Geometry Measurements for Maintenance Planning
84 TRANSPORTATION RESEARCH RECORD 147 Use of Trak Geometry Measurements for Maintenane Planning WILLEM EBERSOHN AND ERNEST T. SELIG Trak geometry measurements are disussed as a means of evaluating the
More informationIEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 9, NO. 3, MAY/JUNE 2012 401
IEEE TRASACTIOS O DEPEDABLE AD SECURE COMPUTIG, VOL. 9, O. 3, MAY/JUE 2012 401 Mitigating Distributed Denial of Servie Attaks in Multiparty Appliations in the Presene of Clok Drifts Zhang Fu, Marina Papatriantafilou,
More informationPath Planning of Automated Optical Inspection Machines for PCB Assembly Systems
96 International Journal TaeHyoung of Control, Park, Automation, HwaJung Kim, and Systems, and Nam vol. Kim 4, no., pp. 9604, February 2006 Path Planning of Automated Optial Inspetion Mahines for PCB
More informationSpecial Relativity and Linear Algebra
peial Relativity and Linear Algebra Corey Adams May 7, Introdution Before Einstein s publiation in 95 of his theory of speial relativity, the mathematial manipulations that were a produt of his theory
More information5.2 The Master Theorem
170 CHAPTER 5. RECURSION AND RECURRENCES 5.2 The Master Theorem Master Theorem In the last setion, we saw three different kinds of behavior for reurrenes of the form at (n/2) + n These behaviors depended
More informationREDUCTION FACTOR OF FEEDING LINES THAT HAVE A CABLE AND AN OVERHEAD SECTION
C I E 17 th International Conferene on Eletriity istriution Barelona, 115 May 003 EUCTION FACTO OF FEEING LINES THAT HAVE A CABLE AN AN OVEHEA SECTION Ljuivoje opovi J.. Elektrodistriuija  Belgrade 
More informationAgentBased Grid Load Balancing Using PerformanceDriven Task Scheduling
AgentBased Grid Load Balaning Using PerformaneDriven Task Sheduling Junwei Cao *1, Daniel P. Spooner, Stephen A. Jarvis, Subhash Saini and Graham R. Nudd * C&C Researh Laboratories, NEC Europe Ltd.,
More information