Average-Case Analysis of a Nearest Neighbor Algorithm. Moett Field, CA USA. in accuracy. neighbor.

Transcription

1 From Proceeings of the Thirteenth International Joint Conference on Articial Intelligence (1993). Chambery, France: Morgan Kaufmann Average-Case Analysis of a Nearest Neighbor Algorithm Pat Langley Learning Systems Department Siemens Corporate Research 755 College Roa East Princeton, NJ USA Wayne Iba AI Research Branch Mail Stop NASA Ames Research Center Moett Fiel, CA USA Abstract In this paper we present an average-case analysis of the nearest neighbor algorithm, a simple inuction metho that has been stuie by many researchers. Our analysis assumes a conjunctive target concept, noise-free Boolean attributes, an a uniform istribution over the instance space. We calculate the probability that the algorithm will encounter a test instance that is istance from the prototype of the concept, along with the probability that the nearest store training case is istance e from this test instance. From this we compute the probability of correct classication as a function of the number of observe training cases, the number of relevant attributes, an the number of irrelevant attributes. We also explore the behavioral implications of the analysis by presenting preicte learning curves for articial omains, an give experimental results on these omains as a check on our reasoning. 1. Nearest Neighbor Algorithms Most learning methos form some abstraction from experience an store this structure in memory. The el has explore a wie range of such structures, incluing ecision trees (Quinlan, 1986), multilayer networks (Rumelhart & McClellan, 1986), an probabilistic summaries (Fisher, 1987). However, in recent years there has been growing interest in methos that store instances or cases in memory, an that apply this specic knowlege irectly to new situations. This approach goes by many names, incluing instance-base learning an case-base reasoning, an one can apply it to many ierent tasks. The simplest an most wiely stuie class of techniques, often calle nearest neighbor algorithms, originate in the el of pattern recognition (Cover & Hart, 1967; Dasarathy, 1991) an applies to classication tasks. In the basic metho, learning appears almost trivial { one simply stores each training instance in memory. The power of the metho comes from the retrieval process. Given a new test instance, one ns the store training case that is nearest accoring to some istance measure, notes the class of the retrieve case, an preicts the new instance will have the same class. Many variants exist on this basic algorithm. For instance, Stanll an Waltz (1986) have stuie a version that retrieves the k closest instances an bases preictions on a weighte vote, incorporating the istance of each store instance from the test case; such techniques are often referre to as k-nearest neighbor algorithms. Others (Cover & Hart, 1967; Aha, Kibler, & Albert, 1991) have stuie an alternative approach that stores cases in memory only upon making an error, thus reucing memory loa an retrieval time with little reuction in accuracy. We woul like to unerstan the learning behavior of this intriguing class of methos uner various conitions. Aha et al. present a PAC analysis of one such algorithm, but our aim is to obtain tighter bouns that we can irectly relate to experimental results. To this en, we ecie to pursue an average-case analysis along the lines evelope by Hirschberg an Pazzani (1991) for logical inuction methos an by Langley, Iba, an Thompson (1992) for probabilistic ones. For the sake of tractability, we focuse our eorts on the most basic of the instance-base techniques, which stores all training cases an bases its preiction on the single nearest neighbor. However, the simplicity of this metho oes not mean it lacks power. Aha et al. (1991) report the results of an experimental stuy that compare the algorithm (which they calle IB1 ) to Quinlan's (1986) more sophisticate C4 algorithm for inucing ecision trees. Table 1 contains the results on four natural omains, two of them (\Clevelan" an \Hungarian") involving preiction of heart isease from symptoms, another concerning the iagnosis of primary tumors, an a fourth involving preiction of party aliations for members of Congress from their voting recors. For each omain, Aha et al. traine the algorithms on approximately 80% of the cases an teste them on the remaining instances, averaging over 50 ierent partitions. On the Clevelan ata, the two algorithms' performance was inistinguishable, an IB1's behavior on the tumor an voting recors nearly reache C4's level. Although the basic nearest neighbor algorithm fare much worse on the Hungarian ata set, simple moications prouce accuracy comparable to that for C4 (Aha, 1990), an its performance on the other omains argues that it eserves closer inspection in any case. In the remainer of this paper, we report the initial results of our average-case analysis of the simple nearest neighbor metho. We begin by presenting the as-

2 Analysis of a Nearest Neighbor Algorithm 890 Table 1. Percentage accuracies for a nearest neighbor metho (IB1) an a ecision-tree algorithm (C4) on four classication omains, taken from Aha et al. (1991). Each column reports both average accuracy of classication an stanar error. Domain IB1 C4 Clevelan Hungarian Primary tumor Voting recors sumptions of the analysis, followe by our erivation of the equations for preicting accuracy as a function of three variables. After this, we examine the implications of the analysis for the algorithm's behavior, comparing preicte learning curves for articial omains of iering iculty. Finally, we note some limitations of the analysis an suggest irections for future research. 2. An Average-Case Analysis Recall that the algorithm uner stuy stores all training cases in memory. Upon encountering a test instance, it retrieves the nearest observe case an preicts the same class as that for the store instance. If a tie occurs, the version we will examine selects one of the nearest cases at ranom. We woul like to compute A n, the probability of correct classication (i.e., the preictive accuracy) after n training instances as a function of characteristics of the omain. Our analysis will assume that there exist two classes, C an C, ene over r relevant Boolean attributes A j an i irrelevant ones. We will also assume that the probability of occurrence P (A j ) = 1=2 for each such attribute A j, generating a uniform istribution over the instance space, an that the target concept is conjunctive, giving P (C) = P (A) r = 1=2 r. Finally, we will measure the istance between two cases as the number of attributes on which they ier. Because there are r + i attributes, there are exactly r+i+1 istinct istances, ranging from zero to r + i, that can occur. We will use I to enote an arbitrary test case that is istance from the prototype P for the positive class. We will also refer to J e as an arbitrary training case that is istance e from test instance I. We will often treat groups of test cases as a class base on their istance from the prototype; similarly, we will consier groups of training instances base on their istance e from a given class of test cases. Our strategy consiers positive an negative test instances separately. We will use A(C) n to refer to the probability of correct classication given that the algorithm encounters a positive test case after n training cases, an we will use A( C)n for the preictive accuracy given a negative test instance. The overall accuracy of the nearest neighbor algorithm after n training cases is A n = P (C)A(C) n + P ( C)A( C)n ; (1) where P (C) is the probability of a positive test case an P ( C) = 1? P (C) is the probability of a negative one. Let us eal with the accuracy on negative test cases rst. We can view this term as a weighte sum of the accuracies for ierent types of test cases I, where is the istance from the prototype. 1 We must sum over all possible istances from the prototype at which a test case can occur (except for = 0), multiplying the probability of each type by its accuracy B( C) ;n accoring to the possible contents of memory after n training instances: A( C)n = Xr+i =1 where the accuracy component? 2 r+i? 2 i B( C);n ; (2) B( C);n = N(J?i?1) n +T (J?i;+i)? n +F (J +i+1 ) n (3) can be further ivie into three terms, which are relate to the regions shown in Figure 1. These correspon, respectively, to situations in which the closest store training case is near enough to I to ensure correct classication, in which ties can occur because the closest case may belong to either class, an in which the closest instance is far enough from I to ensure correct preiction. The initial term, N(J?i?1) n, represents the contribution to the accuracy that results from the rst of these situations, when the nearest store training case J e is closer to I than the latter is to the nearest positive instance (i.e., 0 e < ). When this occurs, the algorithm will correctly classify I as negative. To see this, consier the innermost region in Figure 1, an note that the algorithm must observe only one training case within the region for this to transpire. The probability that any given training instance will fall within e or fewer steps of the test case is W (J e ) = 1 ex r + i ; (4) 2 r+i j j=0 giving the probability 1?W (J e ) that this will not occur. Thus, the probability that (after n training instances) the algorithm will have seen at least one such case is N(J e ) n = 1? [1? W (J e )] n ; (5) which gives the rst term in the enition of B( C);n in equation (3). 1. When some features are irrelevant, there are multiple positive instances, any of which we can select as the prototype without loss of generality.

3 Analysis of a Nearest Neighbor Algorithm i r + i I i P We can further ecompose the rst term in the prouct into the probability that exactly k of the n training cases are istance e from the test case an that the remaining n? k are at some greater istance (since the k cases are the nearest ones), giving " # k n e M(J e ) n;k = [1 k 2 r+i? W (J e )] n?k (7) as the formal expression. To compute the accuracy given a tie among k store cases, we must sum over the possible numbers j of negative instances, in each case multiplying the subaccuracy by the probability of that occurrence. This gives E(J e )? k = kx j=0 j k k [1? V (C)? e; j ]j [V (C)? e; ]k?j ; (8) Figure 1. Regions of interest for computing the probability of correctly classifying a negative test case I, istance from prototype P, using the store training case J e, which is istance e from I. The inner region epicts the probability that e <, the outer region shows the probability that e > + i, an the central region inicates the probability that e + i. The secon term from this equation, T (J?i;+i)? n, is the contribution to the accuracy when the istance to the nearest store training instance J e is between the istances to the nearest possible positive instance an the farthest one (i.e.,?i e +i). This correspons to the central region in the gure. In this situation, con- icts can occur uring the classication process, in that the algorithm may retrieve both positive an negative training cases at istance e from the test case. Ties are possible in this region because, given i irrelevant attributes, there are 2 i positive instances that can be locate i steps or less away from the prototype. For any given test case I that is istance from the prototype, the nearest store positive case may be i steps away from the prototype in the irection towar I, i steps away from the prototype in the irection away from I, or somewhere between these two extremes. Negative instances can also occur anywhere within this region, making the entire mile ban in the gure open to the possibility of ties. To hanle all possible ties, we must sum over all istances e between an + i, then sum over the possible numbers k of nearest instances (positive or negative) that have been store at each such istance. In each case, we must multiply the probability M(J e ) n;k of that number occurring by the accuracy E(J e )? k that results from such a tie. We can state this formally as T (J a;b )? n = bx nx e=a k=1 M(J e ) n;k E(J e )? k : (6) where j=k is the expecte accuracy when one selects a training case at ranom from a set that contains j out of k negative instances. The term V (C)? e; represents the probability of a positive instance J e given that the instance is e steps away from negative test case I, which is in turn steps away from the prototype. We can expan this term to V (C)? min(r;) X e; = k=1? r i k??k? e?k e ; (9) provie k e, k r, an? k i, an to zero otherwise. This expression sums over ierent ways in which istance can occur, with some steps along relevant attributes an others along irrelevant ones, multiplie by the probability that an instance e steps away from the resulting test case will be positive. The nal term from equation (3), F (J +i+1 ) n, species the contribution to the accuracy when the nearest training instance J e has istance greater than + i from test case I (i.e., + i < e r + i), an thus falls within the outermost region in Figure 1, which contains only negative instances. As with the N(J?i?1) n term, the algorithm will correctly classify I as negative whenever this occurs. The chance that this situation will arise is F (I e ) n = Xr+i 2 r+i j=e r + i j 3 5n ; (10) which is the probability that any given training case will fall at istance e or greater from the test instance, taken to power n to generate the probability that every training instance seen so far satises this conition. Now we can turn to A(C) n, the accuracy on positive test cases after n training instances. The situation here is simpler than for negative test cases, but still nontrivial. The algorithm is guarantee to classify a positive test case I correctly only when the nearest store training instance is itself the test case (i.e., e = 0). Ties can occur anywhere in the range 1 e i, giving the expression

4 Analysis of a Nearest Neighbor Algorithm 892 j=0 A(C) n = N(J 0 ) n + T (J 1;i ) + n : (11) Because one can treat any positive instance as the prototype, there is no nee to sum over ierent istances here. Moreover, since no positive instance can be more than i steps away from any other, we can omit the thir term of equation (3), F (J i+1 ) n, which is always zero. The term for hanling ties is analogous to equation (6) for the negative situation, but we must revise the enition for E(J e )? k in equation (8) to kx E(J e ) + k = k? j k [1? V (C) + e ] j [V (C) + e ] k?j : (12) k j Note that here we must use the numerator k? j rather than k, in that we are ealing with positive test cases. Moreover, we must take a ierent approach to computing V (C) + e, the probability of a positive instance J e given that the instance is e steps away from positive test case I. In this case, we have V (C) + e = e e ; (13) when i e, but zero in other situations. Taken together, the enitions for A(C) n, A( C) n, an their component terms let us preict the overall accuracy A n for the nearest neighbor algorithm as a function of the number of training instances n, the number of relevant attributes r, an the number of irrelevant attributes i. 3. Behavioral Implications of the Analysis Although the equations in the previous sections provie a formal characterization of the nearest neighbor algorithm's behavior, their implications are not obvious. To better unerstan the eects of omain characteristics, we systematically varie certain omain parameters an examine the preicte results. In aition to computing theoretical preictions, we also collecte experimental learning curves that summarize the algorithm's actual behavior. Each atum on these curves reports the classication accuracy average over 100 runs on ranomly generate training sets, measure over the entire space of uniformly istribute noise-free instances. In each case, we boun the mean accuracy with 95% conence intervals to show the egree to which our preicte learning curves t the observe ones. These experimental results provie an important check on our reasoning, an they ientie a number of problems uring evelopment of the analysis. Figure 2 shows the eects of the number of relevant attributes in the conjunctive target concept. For this stuy, we hel the number of irrelevant attributes i constant at one, an we varie both the number of training instances an the number of relevant attributes r. As typical with learning curves, the accuracy starts low an graually improves as the algorithm encounters more training instances. The eects of target complexity also make sense. Increasing the number of relevant features Probability of correct classification relevant 2 relevants 4 relevants Number of training instances Figure 2. Preictive accuracy of the nearest neighbor algorithm on a conjunctive concept, assuming the presence of one irrelevant attribute, as a function of training instances an the number of relevant attributes. The lines represent theoretical learning curves, whereas the error bars inicate experimental results. shoul increase the overall number of negative instances, giving higher accuracy early in the inuction process; however, this factor also increases the total number of possible instances, requiring more training cases to reach asymptote an proucing a crossover eect. The learning rate seems to egrae gracefully with increasing complexity, an the theoretical an actual learning curves are in close agreement, which lens conence to the analysis. The sensitivity of the nearest neighbor algorithm to irrelevant attributes is more ramatic, as shown in Figure 3. This graph summarizes the results of a similar stuy of the interaction between the number of training instances n an the number of irrelevant attributes. Here we hel the number of relevant attributes constant at two, an we examine three levels of the i parameter. As with the previous stuy, the egraation in learning rate is graceful, but the eect is somewhat greater. The ierence between the two results appears more signicant when one realizes that increasing i oes not reuce the proportion of positive instances, as oes increasing the number of relevant attributes. These observations are consistent with Aha's (1990) reports on the sensitivity of nearest neighbor methos to the number of irrelevant attributes. We can also compare the behavior of the nearest neighbor algorithm to that of other inuction methos for which average-case analyses exist. In particular, Pazzani an Sarrett (1992) have stuie the Wholist algorithm, which initializes its concept escription to the conjunction of features in the rst positive training instance, then removes any feature that fails to occur in later positive instances. Similarly, Langley, Iba, an Thompson (1992) have analyze the behavior of the Bayesian classier, a simple probabilistic metho that stores observe

5 Analysis of a Nearest Neighbor Algorithm 893 Probability of correct classification irrelevant 3 irrelevants 5 irrelevants Training instances require Nearest neighbor Bayesian classifier Number of training instances Number of irrelevant attributes Figure 3. Preictive accuracy of the nearest neighbor algorithm on a conjunctive concept, assuming the presence of two relevant attributes, as a function of training instances an the number of irrelevant attributes. The lines represent theoretical learning curves, whereas the error bars inicate experimental results. base rates an conitional probabilities. As Pazzani an Sarrett note, Wholist's learning rate is unaecte by the number of relevant attributes, so their algorithm clearly scales up better on this imension than oes the nearest neighbor technique. Comparison to the Bayesian classier on this factor is more icult, in that Langley et al.'s stuy examine equal probabilities for the two classes, whereas the current analysis assumes that they ier. In some omains, eective learning relies more on the ability to hanle many irrelevant features than many relevant ones. In this vein, we have shown analytically that the number of training instances require for Wholist to achieve a given level of accuracy increases only with the logarithm of the number of irrelevant attributes. Although we have not yet erive similar analytic relations for the nearest neighbor or probabilistic methos, we can use the existing analyses to estimate ability to scale on this imension. Figure 4 graphs the preicte number of training instances neee to achieve 90% accuracy for each algorithm as a function of the number of irrelevant attributes, assuming a target concept involving only one relevant feature an a uniform istribution of instances. The analyses o not provie these quantities irectly, but one can interpolate them from the theoretical learning curves. The gure reveals that the Bayesian classier scales well to increasing numbers of irrelevant attributes, with the epenent measure growing as an approximate linear function of this factor. In contrast, the number of training instances require by the nearest neighbor metho grows much faster, although we cannot yet etermine the precise superlinear relation. These results are also consistent with Aha's (1990) conclusions about the response of stanar instance-base methos to many irrelevant features. Figure 4. Theoretical number of training instances (interpolate) require to reach 90% accuracy by a nearest neighbor algorithm an a simple Bayesian classier on a conjunctive concept, assuming the presence of one relevant attribute, as a function of the number of irrelevant attributes. However, the above comparisons are not entirely fair. Neither the Wholist algorithm nor the Bayesian classier are esigne to hanle isjunctive concepts, which present no obstacles to even the simplest nearest neighbor algorithm. Our focus on conjunctive concepts in the current analysis has obscure this strength. Also, Aha (1990) has evelope a variant of the nearest neighbor algorithm that retains statistics on the usefulness of each attribute, an he has shown that this approach fares better in omains with many irrelevant terms. Nevertheless, the ability to make comparisons of the above type is one avantage of careful formal analyses, an they have provie insights about the relative strengths of the ierent learning algorithms. 4. General Discussion In this paper we presente an average-case analysis of the most basic nearest neighbor algorithm. Our treatment assumes that the target concept is conjunctive, that instances are free of noise, that attributes are Boolean, an that instances are uniformly istribute. Given information about the number of relevant an irrelevant attributes, our equations let us compute the expecte classication accuracy after a given number of training instances. To explore the implications of the analysis, we plotte the preicte behavior of the algorithm as a function of these three factors, ning graceful egraation as the number of relevants r an irrelevants i increase, but ning a stronger eect for the secon. As a check on our analysis, we ran the algorithm on articial omains with the same characteristics. The preicte behavior closely t that foun in the experiments, but only after correcting several errors in our reasoning that the empirical stuies reveale.

6 Analysis of a Nearest Neighbor Algorithm 894 These results begin to account for the wie range of performance observe for the algorithm by Aha an others on natural omains. However, a full explanation will require several extensions to the analysis. In particular, we must incorporate the inuence of both class an attribute noise, as we have one in earlier analyses (Iba & Langley, 1992; Langley et al., 1992). We must also hanle situations in which each attribute follows a separate probability istribution, following the approach taken by Hirschberg an Pazzani (1991). Even more important, we must exten the framework to hanle broaer classes of target concepts. Nearest neighbor methos are well suite for M of N concepts, in which any M of the N features in the prototype are sucient for membership in the class. Since istance from the prototype plays a central role in the current analysis, we believe extening it to hanle such concepts will be quite feasible. Similarly, because the algorithm stores many training instances in memory, it can easily acquire isjunctive concepts that require multiple prototypes. Again, we hope that simple extensions to the existing framework will hanle this situation. We shoul also generalize the analysis to inclue k-nearest neighbor methos, following the lea recently provie by Turney (in press). Another irection for future work woul attempt to map the extene analysis onto natural omains in which there alreay exist experimental results with the metho. Given information about the istributions of attributes (which are available in the ata), along with estimates of the noise levels an target concepts (which require informe guesses), we can compare learning curves preicte by the theory with those observe in experimental runs. This approach woul exten the applicability of our average-case moel beyon the articial omains to which we have limite our tests to ate. In summary, we believe that our initial analysis has provie some useful insights about the behavior of the basic nearest neighbor algorithm. These begin to explain why the algorithm compares favorably with more complex inuction methos on some omains but not others, an our results are consistent with intuitions about the algorithm's sensitivity to irrelevant attributes. We also believe the existing theoretical framework can be extene to hanle more challenging target concepts an other factors that complicate the learning task, thus proviing a soli base on which to carry out further stuies of instance-base learning. Acknowlegements We woul like to thank Stephanie Sage an Davi Aha for iscussions that helpe clarify our ieas, Davi Aha an three reviewers for useful comments, an Nils Nilsson for his support an encouragement. The rst author contribute to this work while he was at the Center for the Stuy of Language an Information, Stanfor University, an the Institute for the Stuy of Learning an Expertise. References Aha, D. W. (1990). A stuy of instance-base algorithms for supervise learning tasks: Mathematical, empirical, an psychological evaluations. Doctoral issertation, Department of Information & Computer Science, University of California, Irvine. Aha, D. W., Kibler, D., & Albert, M. K. (1991). Instancebase learning algorithms. Machine Learning, 6, 37{ 66. Cover, T. M., & Hart, P. E. (1967). Nearest neighbor pattern classication. IEEE Transactions on Information Theory, 13, 21{27. Dasarathy, B. V. (E.). (1991). Nearest neighbor (NN) norms: NN pattern classication techniques. Los Alamitos, CA: IEEE Computer Society Press. Fisher, D. H. (1987). Knowlege acquisition via incremental conceptual clustering. Machine Learning, 2, 139{172. Hirschberg, D. S., & Pazzani, M. J. (1991). Averagecase analysis of a k-cnf learning algorithm (Technical Report 91{50). Irvine: University of California, Department of Information & Computer Science. Iba, W., & Langley, P. (1992). Inuction of one-level ecision trees. Proceeings of the Ninth International Conference on Machine Learning (pp. 233{240). Abereen, Scotlan: Morgan Kaufmann. Langley, P., Iba, W., & Thompson, K. (1992). An analysis of Bayesian classiers. Proceeings of the Tenth National Conference on Articial Intelligence (pp. 223{ 228). San Jose, CA: AAAI Press. Pazzani, M. J., & Sarrett, W. (1992). A framework for the average case analysis of conjunctive learning algorithms. Machine Learning, 9, 349{372. Quinlan, J. R. (1986). Inuction of ecision trees. Machine Learning, 1, 81{106. Rumelhart, D. E., & McClellan, J. L. (1986). On learning the past tenses of English verbs. In J. L. McClellan & D. E. Rumelhart (Es.), Parallel istribute processing: Explorations in the microstructure of cognition (Vol. 2). Cambrige, MA: MIT Press. Stanll, C., & Waltz, D. (1986). Towar memory-base reasoning. Communications of the ACM, 29, 1213{ Turney, P. (in press). Voting in instance-base learning: A theoretical analysis. Journal of Experimental an Theoretical Articial Intelligence.