Learning From Small Samples: An Analysis of Simple Decision Heuristics

Transcription

1 Learning From Small Samles: An Analysis of Simle Decision Heuristics Özgür Şimşek and Marcus Buckmann Center for Adative Behavior and Cognition Max Planck Institute for Human Develoment Lentzeallee 9, 119 Berlin, Germany {ozgur, Abstract Simle decision heuristics are models of human and animal behavior that use few ieces of information erhas only a single iece of information and integrate the ieces in simle ways, for examle, by considering them sequentially, one at a time, or by giving them equal weight. We focus on three families of heuristics: single-cue decision making, lexicograhic decision making, and tallying. It is unknown how quickly these heuristics can be learned from exerience. We show, analytically and emirically, that substantial rogress in learning can be made with just a few training samles. When training samles are very few, tallying erforms substantially better than the alternative methods tested. Our emirical analysis is the most extensive to date, emloying natural data sets on diverse subjects. 1 Introduction You may remember that, on January 1, 2009, in New York City, a commercial assenger lane struck a flock of geese within two minutes of taking off from LaGuardia Airort. The lane immediately and comletely lost thrust from both engines, leaving the crew facing a number of critical decisions, one of which was whether they could safely return to LaGuardia. The answer deended on many factors, including the weight, velocity, and altitude of the aircraft, as well as wind seed and direction. None of these factors, however, are directly involved in how ilots make such decisions. As coilot Jeffrey Skiles discussed in a later interview [1], ilots instead use a single iece of visual information: whether the desired destination is staying stationary in the windshield. If the destination is rising or descending, the lane will undershoot or overshoot the destination, resectively. Using this visual cue, the flight crew concluded that LaGuardia was out of reach, deciding instead to land on the Hudson River. Skiles reorted that subsequent simulation exeriments consistently showed that the lane would indeed have crashed before reaching the airort. Simle decision heuristics, such as the one emloyed by the flight crew, can rovide effective solutions to comlex roblems [2, ]. Some of these heuristics use a single iece of information; others use multile ieces of information but combine them in simle ways, for examle, by considering them sequentially, one at a time, or by giving them equal weight. Our work is concerned with two questions: How effective are simle decision heuristics? And how quickly can they be learned from exerience? We focus on roblems of comarison, where the objective is to decide which of a given set of objects has the highest value on an unobserved criterion. These roblems are of fundamental imortance in intelligent behavior. Humans and animals send much of their time choosing an object to act on, with resect to some criterion whose value is unobserved at the time. Choosing a mate, a rey to chase, an investment strategy for a retirement fund, or a ublisher for a book are just a few examles. Earlier studies on this roblem have shown 1

2 that simle heuristics are surrisingly accurate in natural environments [,,,,, 9], esecially when learning from small samles [, ]. We resent analytical and emirical results on three families of heuristics: lexicograhic decision making, tallying, and single-cue decision making. Our emirical analysis is the most extensive to date, emloying natural environments on diverse subjects. Our main contributions are as follows: (1) We resent analytical results on the rate of learning heuristics from exerience. (2) We show that very few learning instances can yield effective heuristics. () We emirically investigate single-cue decision making and find that its erformance is remarkable. () We find that the most robust decision heuristic for small samle sizes is tallying. Collectively, our results have imortant imlications for develoing more successful heuristics and for studying how well simle heuristics cature human and animal decision making. 2 Background The comarison roblem asks which of a given set of objects has the highest value on an unobserved criterion, given a number of attributes of the objects. We focus on airwise comarisons, where exactly two objects are being comared. We consider a decision to be accurate if it selects the object with the higher criterion value (or either object if they are equal in criterion value). In the heuristics literature, attributes are called cues; we will follow this custom when discussing heuristics. The heuristics we consider decide by comaring the objects on one or more cues, asking which object has the higher cue value. Imortantly, they do not require the difference in cue value to be quantified. For examle, if we use height of a erson as a cue, we need to be able to determine which of two eole is taller but we do not need to know the height of either erson or the magnitude of the difference. Each cue is associated with a direction of inference, also known as cue direction, which can be ositive or negative, favoring the object with the higher or lower cue value, resectively. Cue directions (and other comonents of heuristics) can be learned in a number of ways, including social learning. In our analysis, we learn them from training examles. Single-cue decision making is erhas the simlest decision method one can imagine. It comares the objects on a single cue, breaking ties randomly. We learn the identity of the cue and its direction from a training samle. Among the 2k ossible models, where k is the number of cues, we choose the cue, direction combination that has the highest accuracy in the training samle, breaking ties randomly. Lexicograhic heuristics consider the cues one at a time, in a secified order, until they find a cue that discriminates between the objects, that is, one whose value differs on the two objects. The heuristic then decides based on that cue alone. An examle is take-the-best [], which orders cues with resect to decreasing validity on the training samle, where validity is the accuracy of the cue among airwise comarisons on which the cue discriminates between the objects. Tallying is a voting model. It determines how each cue votes on its own (selecting one or the other object or abstaining from voting) and selects the object with the highest number of votes, breaking ties randomly. We set cue directions to the direction with highest validity in the training set. Paired comarison can also be formulated as a classification roblem. Let y A denote the criterion value of object A, x A the vector of attribute values of object A, and y AB = y A y B the difference in criterion values of objects A and B. We can define the class f of a air of objects as a function of the difference in their criterion values: { 1 if yab > 0 f( y AB ) = 1 if y AB < 0 0 if y AB = 0 A class value of 1 denotes that object A has the higher criterion value, 1 that object B has the higher criterion value, and 0 that the objects are equal in criterion value. The comarison roblem is intrinsically symmetrical: comaring A to B should give us the same decision as comaring B to A. That is, f( y AB ) should equal f( y BA ). Because the latter equals f( y AB ), we have the following symmetry constraint: f(z) = f( z), for all z. We can exect better classification accuracy if we imose this symmetry constraint on our classifier. 2

3 Building blocks of decision heuristics We first examine two building blocks of learning heuristics from exerience: assigning cue direction and determining which of two cues has the higher redictive accuracy. The former is imortant for all three families of heuristics whereas the latter is imortant for lexicograhic heuristics when determining which cue should be laced first. Both comonents are building blocks of heuristics in a broader sense their use is not limited to the three families of heuristics considered here. Let A and B be the objects being comared, x A and x B denote their cue values, y A and y B denote their criterion values, and sgn denote the mathematical sign function: sgn(x) is 1 if x > 0, 0 if x = 0, and 1 if x < 0. A single training instance is the tule sgn(x A x B ), sgn(y A y B ), corresonding to a single airwise comarison, indicating whether the cue and the criterion change from one object to the other, along with the direction of change. For examle, if x A = 1, y A =, x B = 2, y B =, the training instance is 1, +1. Learning cue direction. We assume, without loss of generality, that cue direction in the oulation is ositive (we ignore the case where the cue direction in the oulation is neutral). Let denote the success rate of the cue in the oulation, where success is the event that the cue decides correctly. We examine two robabilities, e 1 and e 2. The former is the robability of correctly inferring the cue direction from a set of training instances. The latter is the robability of deciding correctly on a new (unseen) instance using the direction inferred from the training instances. We define an informative instance to be one in which the objects differ both in their cue values and in their criterion values, a ositive instance to be one in which the cue and the criterion change in the same direction ( 1, 1 or 1, 1 ), and a negative instance to be one in which the cue and the criterion change in the oosite direction ( 1, 1 or 1, 1 ). Let n be the number of training instances, n + the number of ositive training instances, and n the number of negative training instances. Our estimate of cue direction is ositive if n + > n, negative if n + < n, and a random choice between ositive and negative if n + = n. Given a set of indeendent, informative training instances, n + follows the binomial distribution with n trials and success robability, allowing us to write e 1 as follows: e 1 = P (n + > n ) P (n + = n ) n ( n = k k= n/2 +1 ) k (1 ) n k + I(n is even) 1 2 ( ) n n/2 (1 ) n/2, n/2 where I is the indicator function. After one training instance, e 1 equals. After one more instance, e 1 remains the same. This is a general roerty: After an odd number of training instances, an additional instance does not increase the robability of inferring the direction correctly. On a new (test) instance, the cue decides correctly with robability if cue direction is inferred correctly and with robability 1 otherwise. Consequently, e 2 = e 1 + (1 )(1 e 1 ). Simle algebra yields the following exected learning rates: After 2k + 1 training instances, with two additional instances, the increase in the robability of inferring cue direction correctly is (2 1)((1 )) k+1 and the increase in the robability of deciding correctly is (2 1) 2 ((1 )) k+1. Figure 1 shows e 1 and e 2 as a function of training-set size n and success rate. The more redictive the cue is, the smaller the samle needs to be for a desired level of accuracy in both e 1 and e 2. This is of course a desirable roerty: The more useful the cue is, the faster we learn how to use it correctly. The figure also shows that there are highly diminishing returns, from one odd training-set size to the next, as the size of the training set increases. In fact, just a few instances make great rogress toward the maximum ossible. The third lot in the figure reveals this roerty more clearly. It shows e 2 divided by its maximum ossible value () showing how quickly we reach the maximum ossible accuracy for cues of various redictive ability. The minimum value deicted in this figure is 0., observed at n = 1. This means that even after a single training instance, our exected accuracy is at least % of the maximum accuracy we can reach. And this value rises quickly with each additional air of training instances.

4 Probability of correctly inferring cue direction (e 1) n Probability of correctly deciding (e 2) n e n Figure 1: Learning cue direction. Learning to order two cues. Assume we have two cues with success rates and q in the oulation, with > q. We exand the definition of informative instance to require that the objects differ on the second cue as well. We examine two robabilities, e and e. The former is the robability of ordering the two cues correctly, which means lacing the cue with higher success rate above the other one. The latter is the robability of deciding correctly with the inferred order. We chose to examine learning to order cues indeendently of learning cue directions. One reason is that eole do not necessarily learn the cue directions from exerience. In many cases, they can guess the cue direction correctly through causal reasoning, social learning, ast exerience in similar roblems, or other means. In the analysis below, we assume that the directions are assigned correctly. Let s 1 and s 2 be the success rates of the two cues in the training set. If instances are informative and indeendent, s 1 and s 2 follow the binomial distribution with arameters (n, ) and (n, q), allowing us to write e as follows: e = P (s 1 > s 2 ) P (s 1 = s 2 ) = 0 j<i n P (s 1 = i)p (s 2 = j) n P (s 1 = i)p (s 2 = i) After one training instance, e is 0.+0.( q), which is a linear function of the difference between the two success rates. If we order cues correctly, a decision on a test instance is correct with robability, otherwise with robability q. Thus, e = e + q(1 e ). Figure 2 shows e and e as a function of and q after three training instances. In general, larger values of, as well as larger differences between and q, require smaller training sets for a desired level of accuracy. In other words, learning rogresses faster where it is more useful. The third lot in the figure shows e relative to the maximum value it can take, the maximum of and q. The minimum value deicted in this figure is 90.9%. If we examine the same figure after only a single training instance, we see that this minimum value is.% (figure not shown). i= After training instances: Probability of correctly ordering (e ) q After training instances: Probability of correctly deciding (e ) q After training instances: e / max(,q) q Figure 2: Learning cue order.

5 Emirical analysis We next resent an emirical analysis of natural data sets, most from two earlier studies [, 1]. Our rimary objective is to examine the emirical learning rates of heuristics. From the analytical results of the receding section, we exect learning to rogress raidly. A secondary objective is to examine the effectiveness of different ways cues can be ordered in a lexicograhic heuristic. The data sets were gathered from a wide variety of sources, including online data reositories, textbooks, ackages for R statistical software, statistics and data mining cometitions, research ublications, and individual scientists collecting field data. The subjects were diverse, including biology, business, comuter science, ecology, economics, education, engineering, environmental science, medicine, olitical science, sychology, sociology, sorts, and transortation. The data sets varied in size, ranging from 1 to 01 objects. Many of the smaller data sets contained the entirety of the oulation of objects, for examle, all 29 islands in the Galáagos archielago. The data sets are described in detail in the sulementary material. We resent results on lexicograhic heuristics, tallying, single-cue decision making, logistic regression, and decision trees trained by CART [1]. We used the CART imlementation in rart [1] with the default slitting criterion Gini, c=0, minslit=2, minbucket=1, and -fold cross-validated cost-comlexity runing. There is no exlicit way to imlement the symmetry constraint for decision trees; we simly augmented the training set with its mirror image with resect to the direction of comarison. For logistic regression, we used the glm function of R, setting the intercet to zero to imlement the symmetry constraint. To the glm function, we inut the cues in the order of decreasing correlation with the criterion so that the weakest cues were droed first when the number of training instances was smaller than the number of cues. Ordering cues in lexicograhic heuristics. We first examine the different ways lexicograhic heuristics can order the cues. With k cues, there are k! ossible cue orders. Combined with the ossibility of using each cue with a ositive or negative direction, there are 2 k k! ossible lexicograhic models, a number that increases very raidly with k. How should we choose one if our to criterion is accuracy but we also want to ay attention to comutational cost and memory requirements? We consider three methods. The first is a greedy search, where we start by deciding on the first cue to be used (along with its direction), then the second, and so on, until we have a fully secified lexicograhic model. When deciding on the first cue, we select the one that has the highest validity in the training examles. When deciding on the mth cue, m 2, we select the cue that has the highest validity in the examles left over after using the first m 1 cues, that is, those examles where the first m 1 cues did not discriminate between the two objects. The second method is to order cues with resect to their validity in the training examles, as take-the-best does. Evaluating cues indeendently of each other substantially reduces comutational and memory requirements but erhas at the exense of accuracy. The third method is to use the lexicograhic model among the 2 k k! ossibilities that gives the highest accuracy in the training examles. Identifying this rule is NP-comlete [1, 1], and it is unlikely to generalize well, but it will be informative to examine it. The three methods have been comared earlier [1] on a data set consisting of German cities [], where the fitting accuracy of the best, greedy, validity, and random ordering was 0., 0., 0.2, and 0.00, resectively. Figure (to anel) shows the fitting accuracy of each method in each of the data sets when all ossible airwise comarisons were conducted among all objects. Because of the long simulation time required, we show an aroximation of the best ordering in data sets with seven or more cues. In these data sets, we started with the two lexicograhic rules generated by the greedy and the validity ordering, ket intact the cues that were laced seventh or later in the sequence, and tested all ossible ermutations of their first six cues, trying out both ossible cue directions. The figure also shows the mean accuracy of random ordering, where cues were used in the direction of higher validity. In all data sets, greedy ordering was identical or very close in accuracy to the best ordering. In addition, validity ordering was very close to greedy ordering excet in a handful of data sets. One exlanation is that a continuous cue that is laced first in a lexicograhic model makes all (or almost all) decisions and therefore the order of the remaining cues does not matter. We therefore also examine the binary version of each data set where numerical cues were dichotomizing around the median (Figure bottom anel). There was little difference in the relative ositions of greedy and otimal ordering excet in one data set. There was more of a dro in the relative accuracy of

6 Fitting accuracy Fitting accuracy Data sets Dichotomized data sets 1 1 Best Aroximate best Greedy ordering Validity ordering Random ordering 1 1 Number of cues Number of cues Figure : Fitting accuracy of lexicograhic models, with and without dichotomizing the cues. the validity ordering, but this method still achieved accuracy close to that of the best ordering in the majority of the data sets. We next examine redictive accuracy. Figure shows accuracies when the models were trained on 0% of the objects and tested on the remaining 0%, conducting all ossible airwise comarisons within each grou. Mean accuracy across data sets was 0. for logistic regression, 0. for CART, 0. for greedy lexicograhic and take-the-best, 0. for single-cue, and 0.2 for tallying. Figure shows learning curves, where we grew the training set one airwise comarison at a time. Two individual objects rovided a single instance for training or testing and were never used again, neither in training nor in testing. Consequently, the training instances were indeendent of each other but they were not always informative (as defined in Section ). The figure shows the mean learning curve across all data sets as well as individual learning curves on 1 data sets. We resent the grahs without error bars for legibility; the highest standard error of the data oints dislayed is in Figure and in Figure. A few observations are noteworthy: (1) Heuristics were indeed learned raidly. (2) In the early art of the learning curve, tallying generally had the highest accuracy. () The erformance of single-cue was remarkable. When trained on 0% of the objects, its mean erformance was better than tallying, 0.9 ercentage oints behind take-the-best, and 1. ercentage oints behind logistic regression. () Take-the-best erformed better than or as well as greedy lexicograhic in most data sets. A detailed comarison of the two methods is rovided below. Validity versus greedy ordering in lexicograhic decision making. The learning curves on individual data sets took one of four forms: (1) There was no difference in any art of the learning curve. This is the case when a continuous cue is laced first: This cue almost always discriminates between the objects, and cues further down in the sequence are seldom (if ever) used. Because greedy and validity ordering always agree on the first cue, the learning curves are identical or nearly so. Twenty-two data sets were in this first category. (2) Validity ordering was better than greedy ordering in some arts of the learning curve and never worse. This category included data sets. () Learning curves crossed: Validity ordering generally started with higher accuracy than greedy ordering; the difference diminished with increasing training-set size, and eventually greedy ordering exceeded validity ordering in accuracy (2 data sets). () Greedy ordering was better than validity or-

7 Accuracy Data sets 9 1 Take-the-best Greedy lexicograhic Tallying Single-cue Logistic regression CART 1 Number of cues Figure : Predictive accuracy when models are trained with 0% of the objects in each data set and tested on the remaining 0%. dering in some arts of the learning curve and never worse ( data sets). To draw these conclusions, we considered a difference to be resent if the error bars (± 2 SE) did not overla. Discussion We isolated two building blocks of decision heuristics and showed analytically that they require very few training instances to learn under conditions that matter the most: when they add value to the ultimate redictive ability of the heuristic. Our emirical analysis confirmed that heuristics tyically make substantial rogress early in learning. Among the algorithms we considered, the most robust method for very small training sets is tallying. Earlier work [] concluded that take-the-best (with undichotomized cues) is the most robust model for training sets with to objects but tallying (with undichotomized cues) was absent from this earlier study. In addition, we found that the erformance of single-cue decision making is truly remarkable. This heuristic has been analyzed [19] by assuming that the cues and the criterion follow the normal distribution; we are not aware of an earlier analysis of its emirical erformance on natural data sets. Our analysis of learning curves differs from earlier studies. Most earlier studies [20,, 21,, 22] examined erformance as a function of number of objects in the training set, where training instances are all ossible airwise comarisons among those objects. Others increased the training set one airwise comarison at a time but did not kee the airwise comarisons indeendent of each other [2]. In contrast, we increased the training set one airwise comarison at a time and ket all airwise comarisons indeendent of each other. This makes it ossible to examine the incremental value of each training instance. There is criticism of decision heuristics because of their comutational requirements. For instance, it has been argued that take-the-best can be described as a simle algorithm but its successful execution relies on a large amount of recomutation [2] and that the comutation of cue validity in the German city task would require 0,2 airwise comarisons just to establish the cue validity hierarchy for redicting city size [2]. Our results clearly show that the actual comutational needs of heuristics can be very low if indeendent airwise comarisons are used for training. A similar result that just a few samles may suffice exists within the context of Bayesian inference [2]. Acknowledgments Thanks to Gerd Gigerenzer, Konstantinos Katsikooulos, Malte Lichtenberg, Laura Martignon, Perke Jacobs, and the ABC Research Grou for their comments on earlier drafts of this article. This work was suorted by Grant SI /1-1 to Özgür Şimşek from the Deutsche Forschungsgemeinschaft (DFG) as art of the riority rogram New Frameworks of Rationality (SPP 11).

8 Mean accuracy Accuracy Training set size Diamond Mileage Fish Salary Training set size Take-the-best Greedy lexicograhic Tallying Single-cue Logistic regression CART Number of data sets Land CPU Obesity Hitter Pitcher Car Bodyfat Lake Infant Contracetion City Accuracy Athlete Take-the-best Greedy lexi. Tallying Single-cue Logistic reg. CART Training set size Figure : Learning curves.

9 References [1] C. Rose. Charlie Rose. Television rogram aired on February, [2] G. Gigerenzer, P. M. Todd, and the ABC Research Grou. Simle heuristics that make us smart. Oxford University Press, New York, [] G. Gigerenzer, R. Hertwig, and T. Pachur, editors. Heuristics: The foundations of adative behavior. Oxford University Press, New York, 20. [] J. Czerlinski, G. Gigerenzer, and D. G. Goldstein. How good are simle heuristics?, ages 9. In [2], [] L. Martignon and K. B. Laskey. Bayesian benchmarks for fast and frugal heuristics, ages In [2], [] L. Martignon, K. V. Katsikooulos, and J. K. Woike. Categorization with limited resources: A family of simle heuristics. Journal of Mathematical Psychology, 2():2 1, 200. [] S. Luan, L. Schooler, and G. Gigerenzer. From ercetion to reference and on to inference: An aroach avoidance analysis of thresholds. Psychological Review, 1():01, 201. [] K. V. Katsikooulos. Psychological heuristics for making inferences: Definition, erformance, and the emerging theory and ractice. Decision Analysis, (1): 29, 20. [9] K. B. Laskey and L. Martignon. Comaring fast and frugal trees and Bayesian networks for risk assessment. In K. Makar, editor, Proceedings of the 9th International Conference on Teaching Statistics, Flagstaff, Arizona, 201. [] H. Brighton. Robust inference with simle cognitive models. In C. Lebiere and B. Wray, editors, AAAI sring symosium: Cognitive science rinciles meet AI-hard roblems, ages American Association for Artificial Intelligence, Menlo Park, CA, 200. [] K. V. Katsikooulos, L. J. Schooler, and R. Hertwig. The robust beauty of ordinary information. Psychological Review, ():9, 20. [] G. Gigerenzer and D. G Goldstein. Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review, ():0 9, 199. [1] Ö. Şimşek. Linear decision rule as asiration for simle decision heuristics. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 2, ages Curran Associates, Inc., Red Hook, NY, 201. [1] L. Breiman, J. Friedman, C. J. Stone, and R. A. Olshen. Classification and regression trees. CRC Press, Boca Raton, FL, 19. [1] T. Therneau, B. Atkinson, and B. Riley. rart: Recursive artitioning and regression trees, 201. R ackage version.1-. [1] M. Schmitt and L. Martignon. On the accuracy of bounded rationality: How far from otimal is fast and frugal? In Y. Weiss, B. Schölkof, and J. C. Platt, editors, Advances in Neural Information Processing Systems 1, ages. MIT Press, Cambridge, MA, 200. [1] M. Schmitt and L. Martignon. On the comlexity of learning lexicograhic strategies. Journal of Machine Learning Research, :, 200. [1] L. Martignon and U. Hoffrage. Fast, frugal, and fit: Simle heuristics for aired comarison. Theory and Decision, 2(1):29 1, [19] R. M. Hogarth and N. Karelaia. Ignoring information in binary choice with continuous variables: When is less more? Journal of Mathematical Psychology, 9(2):, 200. [20] N. Chater, M. Oaksford, R. Nakisa, and M. Redington. Fast, frugal, and rational: How rational norms exlain behavior. Organizational Behavior and Human Decision Processes, 90(1):, 200. [21] H. Brighton and G. Gigerenzer. Bayesian brains and cognitive mechanisms: Harmony or dissonance? In N. Chater and M. Oaksford, editors, The robabilistic mind: Prosects for Bayesian cognitive science, ages Oxford University Press, New York, 200. [22] H. Brighton and G. Gigerenzer. Are rational actor models rational outside small worlds? In S. Okasha and K. Binmore, editors, Evolution and rationality: Decisions, co-oeration, and strategic behaviour, ages 9. Cambridge University Press, Cambridge, 20. [2] P. M. Todd and A. Dieckmann. Heuristics for ordering cue search in decision making. In L. K. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems 1, ages MIT Press, Cambridge, MA, 200. [2] B. R. Newell. Re-visions of rationality? Trends in Cognitive Sciences, 9(1): 1, 200. [2] M. R. Dougherty, A. M. Franco-Watkins, and R. Thomas. Psychological lausibility of the theory of robabilistic mental models and the fast and frugal heuristics. Psychological Review, (1):199 21, 200. [2] E. Vul, N. Goodman, T. L. Griffiths, and J. B. Tenenbaum. One and done? Otimal decisions from very few samles. Cognitive Science, ():99,