Partial-Order Planning Algorithms todomainfeatures. Information Sciences Institute University ofwaterloo

Transcription

1 Relating the Performance of Partial-Order Planning Algorithms todomainfeatures Craig A. Knoblock Qiang Yang Information Sciences Institute University ofwaterloo University of Southern California Comuter Science Deartment 4676 Admiralty Way Waterloo, Ont., Canada N2L 3G Marina del Rey, CA Abstract The AI lanning eld has a long history of introducing yet another search algorithm that is believed to be the best in all domains. Some recent examles are nonlin, tweak and snl. Inthis aer we show, by directly comaring the above three lanners, that the quest for an overall winner is doomed to fail. We rst argue that these three lanners dier in a variety ofways, including methods for termination check, causal-link rotection, and subgoal selection. These dierences entail different search behaviors in terms of factors such as the branching factor, the search deth and the time sent by the algorithm on each search node. Furthermore, the search behavior is closely related to the characteristics of the roblem domain in which a lanner is oerating. In this aer we identify one such domain feature, exressed as the ratio of the number of negative threats to the number of ositive threats. We resent an articial domain where we cancontrol this ratio and show thatin fact the lanners show radically dierent erformance as the ratio is varied in this domain. The imlication of this result for someone imlementing a lanning system is that the most aroriate algorithm will deend on the tyes of roblems to be solved by the lanner. Introduction There has been a great deal of work recently on comaring total-order and artial-order lanning systems [, 9], but little has been done in comaring dierent artial-order lanners themselves. Furthermore, there has been a great deal of seculation on whether a articular artial-order lanner is always better than another. In this aer we focus on three basic lanning algorithms, snl, nonlin and tweak, and demonstrate that their relative erformance deends critically on the domain features. We start by xing articular versions of the three lanners, and analyzing their inherent dierences. On the surface, the This is a revised version of an earlier aer [7] that won the Best Paer Award at the Canadian Articial Intelligence Conference. The rst author is suorted by Rome Laboratory of the Air Force Systems Command and the Defense Advanced Research Projects Agency under contract no. F362-9-C-8. The second author is suorted in art by grants from the Natural Science and Engineering Research Council of Canada, and ITRC: Information Technology Research Centre of Ontario. The views and conclusions contained in this aer are the author's and should not be interreted as reresenting the ocial oinion or olicy of DARPA, RL, NSERC, ITRC, or any erson or agency connected with them. three lanners are quite dierent. However, on a careful examination one can nd that they mainly dier in only a few dimensions, including termination check, subgoal selection, and causal-link rotection. Causal links are introduced during lanning to kee track of the roducer-consumer relationshi among the lan stes. If a causal link is threatened by another ste, then some sort of rotection methods are usually called uon. Among the three lanners, tweak rotects nothing, nonlin rotects against all negative threats, and snl rotects against both negative and ositive threats. The conservative causal-link rotection method used in snl revents the lanner from generating redundant lans and thereby could otentially limit the size of the search sace. However, enforcing the strong rotection has a cost, and in some cases can actually reduce the lanning eciency. Inthis aer, we show that none of the lanners is always a winner. In some domains our lanner based on tweak greatly outerforms both a lanner based on nonlin and snl. In other domains, snl and nonlin erform much better than tweak. The challenge is to identify the features of the domains where each lanner is exected to erform well, so that ractitioners can balance the rotection methods based on the alication domain. In the following sections, we rst review the three algorithms. Then we resent an analysis of the algorithms to identify their relative merits. We also reort on two critical domain features that have the greatest imact on the erformance of the lanners. Finally, we resent emirical results on an articial domain to suort the analysis. 2 Comarison of the Algorithms This section resents the snl, tweak, and nonlin lanning algorithms. First, we resent the snl algorithm based on the algorithm descritions of McAllester and Rosenblitt's Find-Comletion algorithm [8] and Barrett and Weld's POCL algorithm []. We start with this algorithm because we can build on the elegant algorithm descrition and imlementation rovided in revious work. Then, we describe the changes necessary to transform the snl algorithm into algorithms that imlement nonlin [] and tweak [2]. 2. The snl Algorithm In the lanning algorithms that we consider below, we follow the notations used by Barrett and Weld []. A lan is a 3-tule, reresented as hs O Bi, where S is a number of stes, O is a set of ordering constraints, and B the set of variable binding constraints associated with a lan. A ste consists of a set of reconditions, an add list, and a delete list. The binding constraints secify whether two variables can be bound to the same constant or not. The core of snl is the recording of the causal links for why For convenience we will simly refer to them as tweak and nonlin. Planning Agents SIGART Bulletin, Vol. 6, No.

2 asteisintroduced into a lan, and for rotecting that urose. If a ste S i adds a roosition to satisfy a recondition of ste S j, then S i! Sj denotes the causal link. An oerator S k is a threat to S i! Sj if S k can ossibly add or delete a literal q that can ossibly be bound to. For convenience, we also refer to the air (S k S i! Sj) as a threat. In addition, we dene an oerator S k to be a ositive threat to S i! Sj, ifs k can ossibly be between S i and S j, and S k adds a literal q that can ossibly be bound to. Likewise, S k is a negative threat if it can ossibly be between S i and S j, and deletes a literal q that can ossibly be bound to. The following algorithm which is an adatation of McAllester and Rosenblitt's Find-Comletion algorithm [8] and Barrett and Weld's POCL algorithm [], has been shown to be sound, comlete, and systematic (never generates redundant lans). Let the notation codesignate(r) denote the codesignation constraints imosed on a set of variable airs R. For examle, if R = f(x i y i) j i = 2...kg, then codesignate(r) = fx i = y i j i = 2...kg. Similarly, noncodesignate(r) denotes the set of non-codesignation constraints on a set R of variable airs. The arameters of the algorithm are: S=Stes, O=Ordering constraints, B= Binding constraints, G= Goals, T=Threats, and L=Causal links. Algorithm snl(hs O Bi T G L). Termination: If G and T are emty, reort success and sto. 2. Declobbering: A ste s k threatens a causal link s i! sj when it occurs between s i and s j, and it adds or deletes. If there exists a threat t 2 T such that t is a threat between a ste s k and a causal link s i! sj 2 L, then: Remove the threat by adding ordering constraints and/or binding constraints using romotion, demotion, or searation. For comleteness, all ways of resolving the threat must be considered. { Promotion: O = O S fs ks ig, B = B { Demotion: O = O S fs js kg, B = B { Searation: O = O S fs is kg S fs ks jg. Let q be the eect of s k that threatens and let P be the set of binding airs be- S tween q and. B = B, where 2 f j = noncodesignate(s) codesignate(p ; s) where s P ^ s 6= g. 2 Recursive invocation: snl(hs O B i T ; ftg G L) 3. Goal selection: Let be a roosition in G, andlet S need be the ste for which is a recondition. 4. Oerator selection: Let S add be an existing ste, or some new ste, that adds before S need. If no such ste exists or can be added then backtrack. Let L = L S fs add! S need g, S = S S fs add g O = S S O fs add S needg and B = B the set of variable bindings to make S add S add. Finally, udate the goal set: G =(G ;fg) reconditions of S add, if new. For comleteness, all ways of achieving the ste must be considered. 2 The ossible binding constraints are mutually exclusive, since systematicity requires that the search sace is artitioned into non-overlaing arts. 5. Threat identication: Let T = ft j for every ste s k that is a ositive or negative threat to a causal link s i! sj 2 L, t =(s k S i! Sj)g. 6. Recursive invocation: snl(hs O B i T G L ). 2.2 The nonlin Algorithm snl is a descendant ofnonlin [], so the algorithms are quite similar and dier mainly in which threats they rotect against and how they erform searation. These twodif- ferences stem from the added constraints on snl that are used to ensure systematicity. nonlin also rovides some additional caabilities such as hierarchical task-network decomosition, but these caabilities are orthogonal to the oint of this aer and are not considered. The rst change to the snl algorithm is in the threat identication ste. In contrast to snl, only the negative threats are added to the list T : Threat identication: Let T = ft j for every ste s k that is a negative threat to a causal link s i! sj 2 L, t = (s k S i! Sj)g. The second change is that to erform searation, there is no requirement that romotion, demotion and searation are made mutually exclusive. In this case, searation simly entails that one or more of the ossible bindings are forced not to codesignate, but imoses no ordering constraints. Searation: O = O. Let q be the eect of s k that ossibly codesignates with and let S P be the set of binding airs between q and. B = B, where 2f j = noncodesignate(e) where e 2 P g. As we will see in the exerimental results section, the dierences in erformance of goal rotection methods emloyed by snl and nonlin are relatively minor. 2.3 The tweak Algorithm The rimary dierence between tweak and the two revious algorithms is that instead of building exlicit causal links for each condition established by the lanner, tweak uses what is called the Modal Truth Criterion [2] to check the truth of each recondition in the lan. This dierence results in four changes from the snl algorithm and only three changes from the nonlin algorithm. The dierences are in termination, searation, goal selection, and threat identication. Each of these are discussed in turn. Since tweak does not maintain exlicit causal links for each recondition, it must test the truth of all of the reconditions in the lan to determine when the lan is comlete. It does this using the Modal Truth Criterion check [2]. This algorithm takes O(n 3 ) time, as comared with the O() time termination routine of snl. We will refer to the algorithm that imlements the Modal Truth Criterion as mtc. This algorithm returns true if a given lan is comlete and otherwise returns a recondition of some ste in the lan that does not necessarily hold. Termination: If mtc(hs O Bi) is true, reort success and sto. Similar to nonlin, there is no requirement that all of the searation constraints are mutually exclusive. Thus, tweak uses the same method for searation as nonlin. Searation: O = O. Let q be the eect of s k that ossibly codesignates with and let P be the set of binding Planning Agents 2 SIGART Bulletin, Vol. 6, No.

3 airs between q and. B = B S,where 2f j = noncodesignate(e) where e 2 P g. Since tweak does not maintain an exlicit set of causal links, there is no exlicit record of which reconditions much be achieved. Thus, goal section is done using the mtc algorithm. The mtc returns a recondition of a ste in the lan that is not necessarily true. Goal Selection: Let be the recondition of ste S need returned by the mtc rocedure. Finally, unlike both snl and nonlin, tweak makes no attemt to rotect all of the reviously established reconditions against either negative or ositive threats. Tweak does, however, ensure that at each ste all negative threats to the most recently built causal link are removed. However, after a recondition is established and threats are removed, it can be clobbered again. In such a case, tweak will have to re-establish the condition. Threat identication: Let l new = S add! S need, which is the causal link constructed in ste 4. Let T = ftjfor every ste s k that is a negative threat to l new, t =(s k l new)g. As we stated above, the mtc routine for the termination check is more exensive than that for snl. However, this does not mean that tweak is less ecient than snl, since in many cases, tweak will exlore fewer nodes. In the next section, we consider the major factors that aect the search sace, and resent a comlexity analysis of the three algorithms. 3 Analyzing the Algorithms 3. Algorithm Comlexities Let eb be the eective branching factor and ed the eective deth of the search tree. In both algorithms, eb is the maximum number of successor lans generated either after ste 2, or after ste 5, while ed is the maximum number of lan exansions in the search tree from the initial lan state to the solution lan state. Then with a breadth-rst search, the time comlexity of search is O(eb ed T node) where T node is the amount of time sent er node. We next analyze the comlexity of the algorithms by eshing out the arameters eb, ed and T node. In this analysis, let P denote the maximum number of reconditions or eects for a single ste, let N denote the total number of oerators in an otimal solution lan, and let A be either the snl, nonlin, or tweak algorithm. To exand the eective branching factor eb, we rst dene the following additional arameters. We use b new for the number of new oerators found by ste 4 for achieving, b old for the number of existing oerators found by ste 4 for achieving, and r t for the number of alternative constraints to remove one threat. The eective branching factor of search by either algorithm is then eb =maxf( b new + b old A) r ta g since each time the main routine is followed, either ste 2 is executed for removing threats, or ste 3 {6 is executed to build causal links. If ste 2 is executed, r t successor states are generated, but otherwise, ( b new + b old) successor lan states are generated. Next, we exand the eective deth ed. In the solution lan, there are N P numberof( S need ) airs, where is a recondition for ste S need. Let fa be the fraction of the N P airs chosen by ste3. For each air ( S need )chosen by ste 3, ste 5 accumulates a set of threats to remove. Let t A be the number of threats generated by ste 5. Finally, let v be the total number of times any xedair( S need )is chosen by ste 3. Then we have ed A = f A N P t A v A: A summary of the arameters can be found in Table 3.. For snl, each air( S need )must be visited exactly once. Therefore, f snl = and v snl =. Also, snl examines every causal link in the current lan in ste 4. Thus, in the average case, the amount of time er node is half of the total number of links in the solution lan, i.e., N P=2. Thus, the average time comlexity for snl is: O(max( b new + b old snl r tsnl ) N P t snl N P ): nonlin's behaviour is similar to snl in that each air ( S need )must be visited exactly once. Therefore, fnonlin = andv nonlin =. Also similar to snl, nonlin examines every causal link in the current lan in ste 4. The dierence between nonlin and snl is that nonlin resolves only negative threats. This means that in general nonlin will have a smaller t value. The average time comlexity for nonlin is: O(max( b new + b old nonlin r tnonlin ) N P t nonlin N P ) In tweak, f tweak, and can be much smaller than one since tweak does not build exlicit causal links for every recondition. If many reconditions already hold, then the number of chosen reconditions by ste 3 in tweak could be much smaller than the total number of reconditions in the solution lan. Since tweak does not rotect any ast causal links, a recondition can be visited twice. Therefore, v tweak. t tweak, on the other hand, should be much smaller than t snl and t nonlin, since tweak only declobbers for the most recently constructed causal link, and only negative threats are considered. Thus the number of threats is much smaller. Finally, tweak uses MTC to check the correctness of a lan, resulting a comlexity er node to be O((N P ) 3 ). Overall, the comlexity oftweak is: O(max( b new + b old tweak r ttweak ) m T tweak where m = f tweak N P t tweak v tweak and T tweak = (N P ) 3 : In the next section, we discuss how these arameters change with certain domain features. 3.2 Systematicity snl is systematic, which means that no redundant lans are generated in the search sace. In contrast, neither tweak nor nonlin are systematic. However, a lanner that is systematic is not necessarily more ecient. The systematicity roerty reduces the branching factor by avoiding redundant lans. However, systematicity isachieved in snl by rotecting against both the negative and ositive threats, which increases the factor t, amultilicative factor in the exonent. Thus, snl reduces the branching factor at a rice of increasing the deth of search. Therefore, one can get a systematic, but less ecient lanning system. Planning Agents 3 SIGART Bulletin, Vol. 6, No.

4 eb ed T node N P f A v A t A r ta b new b old eective branching factor eective search deth average time er node total number of oerators in a lan total number of reconditions er oerator fraction of ( S need ) airs examined by algorithm A average numberoftimesa( S need ) air is visited by A average number of threats found by A at each node average number of ways to resolve a threat by A average number of new establishers for a recondition average number of existing (or old) establishers for a recondition Table : Parameters used in comlexity analysis. 4 Domain Features and Search Performance The analysis in the revious section can be used to redict the relative erformance of the three lanning algorithms in different tyes of domains. An imortant feature of a domain that determines the relative erformance of any two algorithms is the ratio between the number of ositive threats and number of negative threats. The ratio is an imortant factor in dierentiating the algorithms because the major dierence between any two algorithms is the way they handle ositive and negative threats. Among the three algorithms, tweak only avoids some negative threats, snl rotects against all ositive and negative threats, and nonlin rotects against all negative threats but not the ositive ones. 4. Predictions The major dierence between the algorithms manifest themselves in the execution of Ste, the termination subroutine, and Ste 4, threat detection. To see their eect on search eciency, let t + denote the average number of ositive threats, and let t ; be the average number of negative threats detected by Ste 4 of snl. LetR denote the ratio of t ; to t +: R = t ; t+. In this section we redict the erformance of the three lanning algorithms based on the value of R. Case : R Since snl resolves all ositive threats, it imoses more constraints on a lan. Thus, on the average an snl lan is more linearly ordered than either a tweak lan or a nonlin lan. A more linearly ordered lan has a smaller number of existing establishing oerators for a given recondition, and thus a smaller branching factor. Thus, the branching factor of snl is likely to be the smallest among the three, and that for tweak is the largest due to its conservative stand in resolving threats. When t + is relatively large, the total numberofthreatst resolved by snl is large, which in turn increases snl's search deth. Also, for both nonlin and snl, a causal link has to be built for every recondition in a lan, a behavior that xes a lower bound on their search deths. With many ositive threats in a lan, a recondition is more likely to be achievable by an existing ste. Therefore tweak will be able to ski many more reconditions comared to nonlin and snl. Thus the search deth of tweak will be much less than both nonlin and snl, and the search deth of nonlin will be smaller than snl because it does not resolve ositive threats. As R decreases below one, the branching factor for tweak and nonlin increase, while the search deth for snl increases. The time comlexity for the former go u olynomially, while for the latter it goes u exonentially. Moreover, the deth of nonlin is greater than the deth of tweak. Therefore, we redict that when R tweak will erform better than nonlin, which in turn will erform better than snl. Case 2: R As with the revious case, the additional constraints imosed by snl and nonlin over tweak imly that snl will have a smaller branching factor then nonlin, and nonlin will have a smaller branching factor than tweak. However, the difference in the number of threats t resolved by tweak, snl, and nonlin will be reduced since there are fewer ositive threats and more negative threats. The reduced number of ositive threats will reduce the deth for snl and nonlin and the increased number of negative threats increases the chance that tweak will be forced to revisit the same recondition/ste air. As a result, the erformance of the dierent lanners could be very close and will deend on deth and branching factors for the roblems being solved. Case 3: R tweak is likely to have the largest branching factor because every time a negative threat occurs, all existing and new oerators are considered as establishers again. This eect increases the factor b old for tweak, resulting in the eective branching factor for tweak being greater than both snl and nonlin. Also due to its resolution of ositive threats, a snl lan is likelytobemorelinearizedthananonlin lan, thus the branching factor of snl will be smaller than nonlin. Each negative threat creates a chance for tweak to revisit the same recondition/ste air. Since in the R case, there is a large number of negative threats, the number of times each recondition is visited, v tweak, islikely to increase. Since tweak is exected to have a larger branching factor and deth greater than both snl and nonlin, when R tweak is exected to erform the worst. snl will outerform nonlin slightly due its smaller branching factor. 4.2 Emirical Results In order to verify our redictions by comaring snl, nonlin and tweak on roblems with dierent ratios of negative and ositive threats, we constructed an articial domain where we could control the value of R. In this domain, each goalcan Planning Agents 4 SIGART Bulletin, Vol. 6, No.

5 be achieved by a sublan of two stes in a linear sequence. Each ste either achieves a goal condition or a recondition of a later ste. The reconditions of the rst ste always hold in the initial state. In addition, we also added extra oerator eects to create threats in lanning. The diculty ofthe roblems in this domain can be increased by increasing the number of goal conditions and the total number of threats. (defste :action A i :recond I i :equals fg :add fp i I i+ if i < n + I if i = n ; and n + > g :delete fi i; if <i<n ; I n; if i = and n ; > g) Number of Unsolved Problems (defste :action A i2 :recond P i :equals fg :add fg i P i+ if i < n + P if i = n ; and n + > g :delete fp i; if <i<n ; P n; if i = and n ; > g) We used this articial domain to run a set of exeriments to comare the erformance of the dierent lanners. In these exeriments we simultaneously varied the number of ositive and negative interactions, such that the total number of interactions remained the same, but the ratio R changes from zero to innity the number of negative interactions increased from to 9 while the number of ositive interactions decreased from 9 to. Below, we resent the results of our emirical tests on dierent oints of the sectrum of as de- ned by the ratio R. In the exeriments, each roblem was run in snl [], a version of nonlin and a version of tweak that were modied from snl. The roblems were solved using a best-rst search on the solution size in order to fairly comare the size of the roblem saces being searched by each system. All the roblems were run on a SUN IPC in Lucid Common Lis with a 2 CPU second time bound. For each value of ratio R, we ran the systems on 2 randomly generated roblems. The oints shown in the grahs below are an average of the 2 roblems. For the grahs showing branching factor and deth, the averages are underestimated for those systems that did not solve all the roblems within the time limit. Figure shows the number of roblems that were solved within the time limit for each of the systems. Desite the fact that some oints on the grah underestimate the actual numbers, the resulting grahs clearly illustrate the tradeos in the dierent design choices in the lanning algorithms Time er Node The time er node result is shown in Figure 2. As can be seen from the gure, as R increases, the time er node for tweak increases slightly, while for snl and nonlin it decreases slightly. The increase in tweak's time er node is as exected, since as R increases, the number of ositive threats decreases. This results in increasingly larger lans. Since tweak's MTC has a comlexity ofo(n 3 ) for a lan with n oerators, the time er node for tweak should increase as n increases. However, the decrease of time er node for both snl and nonlin is unexected. It turns out that the reason for this behaviour is due to the fact that both snl and nonlin use a more conservative method of goal rotection. When R is large, the number of negative threats is large, and the lans generated by snl and nonlin are more constrained or linear as a result. Unlike the MTC, the threat detection routines for Figure : Comarison of the Number of Problems Unsolved Within the Time Limit Average Time er Node Figure 2: Comarison of the Average Time er Node of each of the Algorithms snl and nonlin runs more eciently in this case, leading to a lower cost of threat detection er node Branching Factor The branching factor results are shown in Figure 3. Most of our redictions for branching factors are observable in the gure. For examle, due to its conservative stand in resolving both ositive and negative threats, snl imoses the most constraints onto a lan, and as a results it generally has the lowest branching factor. Also, as the number of negative threats increases, which constrains the ossible lans, the branching factor decreases to one. However, there are a few surrises shown in the gure. When R, we had redicted that tweak would have a larger branching factor than snl and would be similar to nonlin. This rediction cannot observed from the gure. In order to exlain this eect we have broken the branching factor into the two arts described in the analysis, the establishment branching factor and the declobbering branching factor, which are combined to form the overall branching factor. These grahs are shown in Figures 4 and 5. As shown in the grahs, the smaller than exected branching factor for tweak is due to a smaller than exected establishment branching factor. Planning Agents 5 SIGART Bulletin, Vol. 6, No.

6 Average Branching Factor Average Declobbering Branching Factor Figure 3: Comarison of the Average Branching Factor of each of the Algorithms Figure 5: Comarison of the Average Declobbering Branching Factor of each of the Algorithms Average Establishment Branching Factor Average Search Deth Figure 4: Comarison of the Average Establishment Branching Factor of each of the Algorithms Figure 6: Comarison of the Average Deth of each of the Algorithms Careful analysis of the data shows that this discreancy with the redictions is due to the assumtion that the branching factor is uniform across an entire roblem-solving eisode. In fact, where there are many ositive interactions, tweak quickly narrows in on a lan and reduces the establishment branching factor. In contrast, because both snl and nonlin build exlicit causal links and resolve more threats they send more time in the early lan formation stage when the branching factor is higher. Thus overall, snl and nonlin exand a larger art of the search sace that has a large branching factor, while tweak uses it ability to exloit ositive threats to raidly traverse that art of the search sace Deth The comarison of the search deths is shown in Figure 6 and they are as redicted. The only aarent discreancy is that the dierence between snl and tweak should be larger when R. However, the grah is a bit misleading in this case because it includes roblems that could not be solved within the time bound by nonlin and snl andsoit underestimates their search deth. The overall search deth is comosed of a number of factors described in the analysis, which includes the fraction of the reconditions considered, the average number of times each recondition is visited,a and the average number of threats detected by each algorithm. Figure 7 shows the fraction of reconditions considered. This number should be one for both snl and nonlin but again the grahs are distorted by the fact that these two systems did not comlete all of the roblems within the time limit. In that case, there are a number of reconditions of oerators that had not yet been considered. Note that for most of the roblems, tweak only exanded roughly 6-8% of the reconditions and as the roblems had fewer ositive interactions, it was forced to exand more and more of the reconditions. Figure 8 shows the average of number of times each recondition is visited. As redicted, snl and nonlin visit every recondition exactly once, while tweak visits some reconditions more than once. As the number of negative interactions increase, the value for tweak increases because it does not rotect the conditions that have already been achieved. Figure 9 shows the average number of threats detected by each of the systems. The fact that snl detects a much larger number of threats than both nonlin and tweak comes as no surrise. However, the fact that the number of threats detected by nonlin is less than the number detected by tweak when R was not redicted by the analysis. This aears to be due to the fact that the negative threats that nonlin rotects against imose additional ordering constraints on the lan and a more linearly ordered lan has fewer oten- Planning Agents 6 SIGART Bulletin, Vol. 6, No.

7 Fraction of Preconditions Visited Number of Threats Detected Figure 7: Comarison of the Average Fraction of Preconditions Considered by each of the Algorithms Number of Times Each Precondition is Visited Figure 8: Comarison of the Average Number of Times each Precondition is Visited by each of the Algorithms Figure 9: Comarison of the Average Number of Threats Detected by each of the Algorithms Average CPU Time Figure : Comarison of the Average CPU Time of each of the Algorithms tial threats Average CPU Time The average CPU time for solving roblems in the articial domain is shown in Figure. The result ts exactly with our redictions. One thing to note is that no system erforms absolutely the best throughout the entire sectrum dened by R. Another is that although nonlin did well as comared to snl when R is small, it is never signicantly better than snl. In the case where it does outerform snl it is dramatically worse than tweak. This eect should lend credibility to the rotection against ositive threats as used in snl. Although rotection of ositive threats seemed clumsy when R is small, when the number of negative threats is relatively large the rotection method used by snl imoses more constraints on a lan. The resulting lans in snl's search sace are more linear due to the additional constraints. The comutational advantage of dealing with a more linear lan comensates for the loss of eciency due to the rotection of ositive threats. 5 Related Work As we stated in the introduction, little work has been done on comaring dierent artial order lanners. An excetion is the work by Kambhamati[3, 4], who (concurrently with our work) carried out a set of exeriments to test the merits of dierent artial-order lanners. In that work, a air of artial-order lanners MP and MP-I are roosed that build uon snl and nonlin by making use of multile contributors to achieve a recondition. Exeriments in a set of closely related domains were conducted, and the resulting comarison of snl, nonlin, tweak, MP, and MP-I show that MP-I outerforms all of the rest, and that nonlin in one test erformed much better than both snl and tweak(figure 8, [3]). Contrasting Kambhamati's results to ours, we note that the former is based on a small set of xed domains. Our results clearly demonstrate that varying the ratio R of ositive to negative threats exerienced by a lanner, almost any comarison result can be obtained when R the comarison results should be dramatically dierent from that when R. Thus, it is not surrising that there exists domains, with a secic R value, where snl and/or tweak erform worse than nonlin. From this ersective, the work by Kambhamati can be seen as orthogonal to ours while we search for domain features by which to determine the relative erformance of each system, Kambhamati looks for the best lanner on a single oint in the sectrum of features. Kambhamati [5] has also develoed a general framework for Planning Agents 7 SIGART Bulletin, Vol. 6, No.

8 comaring the dierent design choices in a artial-order lanning algorithm. His aer rovides an excellent analysis of the dierent design tradeos, but does not rovide an emirical validation for his conclusions. More recently, we merged the work described here [7] with the design framework develoed by Kambhamati to rovide a comrehensive framework, analysis, and evaluation [6]. 6 Conclusions In summary, wehave resented a detailed comarison of three secic versions of the snl, nonlin, and tweak lanning algorithms. We have classied their dierences along several dimensions, including termination check, subgoal selection and causal-link rotection. In addition, we analyzed how their search erformance can be aected by these dierences. The analysis rovides a foundation for redicting the conditions under which dierent lanning algorithms will erform well. As the results show, the articular versions of snl and nonlin erformed better than our imlementation of tweak in our articial domain when the ratio of negative threats to ositive threats is large, and tweak erforms signicantly better than snl and nonlin in the oosite case. The imlications of these results for someone building a ractical lanning system is that the most aroriate design strategy to choose deends on the characteristics of the roblem being solved. This aer rovides an imortant ste in building useful lanners by identifying a feature of lanning domains that has a major imact on the erformance of dierent lanning algorithms. Our future work will include identifying additional domain features and eventually constructing a framework in which a lanning algorithm can automatically adat to the domain features. [8] David McAllester and David Rosenblitt. Systematic nonlinear lanning. In Proceedings of the Ninth National Conference on Articial Intelligence, Anaheim, CA, 99. [9] Steven Minton, John Bresina, and Mark Drummond. Commitment strategies in lanning: A comarative analysis. In Proceedings of the Twelfth International Joint Conference on Articial Intelligence, Australia, 99. [] Austin Tate. Generating roject networks. In Proceedings of the Fifth International Joint Conference on Arti- cial Intelligence, ages 888{9, Cambridge, MA, 977. References [] Anthony Barrett and Daniel S. Weld. Partial-order lanning: Evaluating ossible eciency gains. Articial Intelligence, 67():7{2, 994. [2] David Chaman. Planning for conjunctive goals. Arti- cial Intelligence, 32(3):333{377, 987. [3] Subbarao Kambhamati. On the utility of systematicity: Understanding tradeos between redundancy and commitment in artial order lanning. In Proceedings of the Thirteenth International Joint Conference onar- ticial Intelligence, Chambery, France, 993. [4] Subbarao Kambhamati. Multi-contributor causal structures for lanning: A formalization and evaluation. Articial Intelligence, Fall, 994. [5] Subbarao Kambhamati. Renement search as a unifying framework for analyzing lanning algorithms. In Fourth International Conference on Princiles of Knowledge Reresentation and Reasoning, Bonn, Germany, 994. [6] Subbarao Kambhamati, Craig Knoblock, and Qiang Yang. Planning as renement search: A unied framework for evaluating the design tradeos in artial order lanning. Technical Reort 94-2, Deartment of Comuter Science and Engineering, Arizona State University, 994. [7] Craig A. Knoblock and Qiang Yang. Evaluating the tradeos in artial-order lanning algorithms. In Proceedings of the Canadian Articial Intelligence Conference, 994. Planning Agents 8 SIGART Bulletin, Vol. 6, No.