STUDYING EXAMPLES AND SOLVING PROBLEMS 2 many problems that are very similar (if not identical) in form to the worked examples, but the students are a

Transcription

1 Studying Examples and Solving Problems: Contributions to Skill Acquisition J. Gregory Trafton HCI Laboratory Naval Research Laboratory Brian J. Reiser School of Education and Social Policy Northwestern University Address Correspondence to: Greg Trafton Naval Research Lab, Code Overlook Av. S.W. Washington, DC Abstract There is little doubt that examples play a major role in acquiring a new skill. How examples improve learning, however, is subject to some debate. Recently, two dierent classes of theories have been proposed to explain why examples are such an eective manner of learning. Example Generalization models suggest that problem solving rules are acquired while studying examples. Knowledge Compilation models, on the other hand, suggest that examples are useful because they guide future problem solving, where the necessary rules are created. General support for the knowledge compilation model was found and tradeos between studying examples and solving problems are discussed. Guidelines for when to study examples and when to solve problems are also presented. Introduction Typical instruction in problem solving domains includes expository text, annotated examples, and problems to solve. Text expositions usually consist of history, terminology, and descriptions of procedures for solving problems. Worked{out examples are presented as problems with the correct answer and a demonstration of the method used to derive it. Problem solving sections consist of

2 STUDYING EXAMPLES AND SOLVING PROBLEMS 2 many problems that are very similar (if not identical) in form to the worked examples, but the students are asked to work out the problems and construct the answers on their own. What is the relationship between these three areas of \textbook learning?" How does studying examples in an in-depth manner improve problem solving? How does solving problems facilitate the understanding process when a later example is studied? In most schools and classrooms, solving problems is emphasized over studying texts or examples. For example, students are given homework and tests containing problems to solve, with little or no concern over the text or the examples. In stark contrast to this approach, a great deal of evidence suggests that examples are a critical component of instructional material (e.g., Anderson, Farrell, & Sauers, 1984; LeFevre, 1987; Mayer, Sims, & Tajika, 1995; Owen & Sweller, 1988; Zhu & Simon, 1987). Recently, a great deal of research has been devoted to determining the role that examples play in the learning process. Much of this research has shown that examples facilitate learning an enormous amount: Students rely heavily on examples in instructional text, focusing more on adapting the method used in an annotated example than on the explanations of a procedure presented in instructional text (LeFevre & Dixon, 1986; VanLehn, 1986). Two dierent models of the benets of examples have been proposed, however knowledge compilation models, in which applying knowledge from examples to actively solve new problems is a critical component of learning (Anderson, 1987; Pirolli, 1991; VanLehn, Jones, & Chi, 1992), and example generalization models, in which learning occurs primarily while studying the examples themselves (Sweller & Cooper, 1985; VanLehn et al., 1992). The Knowledge Compilation View Several theories of learning have been proposed to explain how people acquire skills in a variety of domains. One of the most inuential theories of learning is the ACT series of models, which includes ACT* (Anderson, 1982, 1983, 1987) and more recently ACT{R (Anderson, 1993; This research was part of the rst author's dissertation at Princeton University, and a preliminary report appeared in the Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society (Trafton & Reiser, 1993). We would like to thank Philip N. Johnson-Laird, Paula Raymond, Matthew S. McGlone, Wayne D. Gray, Irv Katz, and the cognitive lunch participants at Princeton University for comments and suggestions on this research. This research was supported in part by grants N J-1125 to Princeton University and N to Northwestern University from the Oce of Naval Research, and by 6.1 funding from the Oce of Naval Research to the Naval Research Laboratory. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the ocial policies, either expressed or implied, of the U. S. Navy. Thanks to Shari Landes, Shannon Knight, Jason Thompson, and Holly Hillman for subject running help, and Doug Merrill for programming help on Experiment 2.

3 STUDYING EXAMPLES AND SOLVING PROBLEMS 3 Anderson & Lebiere, in press). The ACT models are representative of a group of computational models of learning in which knowledge is acquired while solving problems (e.g., Minton, Carbonell, Knoblock, Kuokka, Etzioni, & Gil, 1989; Newell, 1990). ACT{R has two distinct kinds of knowledge: declarative, represented as an associative memory network which contains the facts known by the system; and procedural, represented as a production system. Procedural knowledge enables the system to apply its knowledge and execute behavior to achieve its goals. Because all knowledge starts in declarative form in ACT{R, learning procedural skills consists of converting declarative knowledge into productions. The only way to form a production is through analogy to an elaborated example. Essentially, examples are used by weak methods such as analogy to facilitate problem solving (Anderson, 1987, 1993; Pirolli, 1991; Pirolli & Anderson, 1985). Students study an example or example unit (parts of the example, like a line of an equation in an algebra problem) and encode that information into declarative memory. When a student tries to solve a problem that is similar to an example seen earlier, the declarative memory of the old example is activated. General{purpose productions then interpret the example and set goals to solve the new problem. Finally, mappings are made between the example's structure and the problem's goals and the analogy has succeeded. Once the analogy has succeeded, a production is created which is based on the analogical source. There is no guarantee, however, that this analogical mechanism created an accurate rule only that if the example was mapped properly to the problem the answer may be correct. It should also be noted that learning is not limited to presented examples and their elaborations. Students can, of course, generate their own examples and elaborations and learn from them as well or better than examples and elaborations presented in the text (e.g., Lovett, 1992). This view of skill acquisition has led Anderson and his colleagues to make some interesting predictions concerning the best and most ecient ways to learn. For example, ACT{R predicts that the trajectory taken to solve a problem is completely irrelevant to which productions are formed and, hence, to how much is learned (Anderson, Conrad, & Corbett, 1989, 1993). According to ACT{R, the same amount is learned, regardless of how much oundering occurred. 1 Recall that productions are formed in ACT{R only through analogy. Once a problem has been solved (regardless of the trajectory), it can then be used as an analogical source for future problems. Therefore, if the student solves the problem, the product, or solution, is what is critical to learning, 1 Motivation and frustration may be increased by excessive oundering, and attention or interest to learn may wane, but ACT{R claims that the quality of productions is not directly aected by oundering.

4 STUDYING EXAMPLES AND SOLVING PROBLEMS 4 not the path that is taken to arrive at the solution (Anderson et al., 1993). A direct consequence of this view is that the more problems that are solved (regardless of the regimen), the larger the number of learned productions, because solving more problems allows more opportunity for analogical use of past problem solutions (Anderson et al., 1993, pp. 160{161). Also, with more practice, the existing productions are strengthened (Anderson, 1982, 1987, 1993). We shall refer to these models as Knowledge Compilation models (based on the terminology of Anderson, 1983). These models claim that construction of problem solving knowledge critically relies on applying knowledge to solve problems (Anderson, 1993; Pirolli, 1991). facilitate problem solving, but the only way to form productions is by solving problems. The Example Generalization View Examples may Another group of theorists, on the other hand, claim that the best way to acquire domain specic expertise is by studying examples. 2 We shall refer to these models as Example Generalization models. This view has been expressed most succinctly by Sweller and his colleagues, and by VanLehn and his colleagues. Sweller & Cooper (1985), for example, claim that students can eectively learn procedures by studying annotated examples with minimal instructional text, and this learning can be more eective than unguided problem solving. Sweller and his colleagues argue that solving problems using weak methods requires a great deal of cognitive resources, particularly memory, and that students do not have enough cognitive resources left over to acquire the skill (Owen & Sweller, 1988; Sweller, 1988). Sweller (1988) further argued that cognitive load can be reduced and problem solving schemas can be built more easily by studying examples. When students study examples, eort can be put into understanding the example rather than solving problems, working memory is thereby decreased, and more can be learned. In addition, important relationships are often easier to discern and understand when presented in examples, rather than by solving problems because during problem solving local goals and relationships may swamp the more global relationships (cf. Sweller, 1988). In fact, Sweller goes on to claim that, contrary to both the format of technical textbooks and most current learning theories, problem solving is a hindrance to skill acquisition and should not be used as a learning device (Owen & Sweller, 1988; Sweller, 1988). Note that Sweller does not 2 It should be noted that theorists who advocate the Knowledge Compilation view use the term \production" to refer to well learned domain specic knowledge. Example Generalization theorists, however, do not typically use the term production. Instead, they use terms that focus on dierent knowledge representations, like \rules" or \expert knowledge" or \schemas" or even just \knowledge." We believe that all these dierent types of knowledge can be represented in terms of productions, but will attempt to represent the Example Generalization theorists in their own language, rather than converting their terms into \productions."

5 STUDYING EXAMPLES AND SOLVING PROBLEMS 5 claim that expertise can not be acquired by solving problems, just that studying examples is much more ecient and eective. However, once the skill is acquired, Sweller and his colleagues believe that additional practice at solving problems increases future problem solving performance. Sweller and his colleagues have presented a computational model and several experiments that investigate this issue. For example, Sweller & Cooper (1985, Experiment 3) gave participants algebra problems to solve. The participants were high school students who could solve basic algebra problems, but were not at a high level of prociency. This experiment presented types of examples and problems with which the students were familiar. As instructional material, all participants received an example sheet containing three worked examples. They were to review the examples and explain to the experimenter why each step was taken. After this review phase, participants started an acquisition phase in which they studied examples and/or solved problems. Participants could refer to the review sheet of examples at any time during the acquisition phase. There were four types of problems (all similar but not identical to the problems on the review sheet), each presented twice. There were two conditions: an example condition and a problem solving condition. The participants in the example condition received pairs of worked out examples and problems to solve in which the problem was isomorphic to the example. The participants in the problem solving condition had to solve both members of the pair. Sweller & Cooper (1985) had participants in the example condition solve the second problem because they claimed that participants needed some motivation to study the previous worked out example. They assumed that motivation would be high if participants knew that a similar problem would need to be solved afterwards. Neither condition was allowed to examine the rst problem while working on the second problem. If the solution was incorrect after ve minutes, the correct answer was given by the experimenter. After the acquisition phase, all participants took a posttest which consisted of four problems isomorphic to the acquisition problems. Neither the past problems nor the instructional materials were available during the posttest. There were no dierences between conditions in number of errors made during the acquisition phase, but the example condition solved the posttest problems faster and with fewer errors than the problem solving condition. Thus, the participants who studied examples performed better than the participants who solved those same problems. Sweller & Cooper (1985) claim that the results from this and similar experiments show that learning from examples is superior to learning from solving problems. This experiment clearly shows that one regimen of instruction (studying examples and solving problems) causes more to be learned than another manner of instruction (simply solving a series of

6 STUDYING EXAMPLES AND SOLVING PROBLEMS 6 problems). There are, however, diculties in using this type of experiment as evidence for the view that students learn how to solve problems simply by studying examples. It is unclear, for example, whether the students learned solely from the examples as Sweller & Cooper (1985) believe, or whether the examples are simply being used to facilitate future problem solving during which key learning occurs, as Knowledge Compilation theorists claim. Thus, the experiment demonstrated the benecial aspects of studying examples, but did not demonstrate the source of that benet. Recently, VanLehn, Chi, and their colleagues have proposed a mechanism for learning from examples that is consistent with Sweller's view that knowledge can be acquired best by studying examples. Chi, Bassok, Lewis, Reimann, & Glaser (1989) suggest that the reason that studying examples is benecial is because students may explain the examples to themselves, a process they call self{explanation (see also Chi, de Leeuw, Chiu, & LaVancher, 1994; Pirolli & Bielaczyc, 1989). Chi et al. (1989) suggest that a critical dierence between more and less successful learners is that successful learners explain examples to themselves more fully than do less successful learners. Explaining an example entails elaborating how and why each example line works. Evidently, one of the most important aspects of self{explanation is nding those inferences in an example that are not made explicit (VanLehn & Jones, 1993b). For example, a physics example may make reference to using the cosine without explaining why the cosine is appropriate (as opposed to the sine or tangent). A student who self{explained the example would attempt to ll his or her \knowledge gap" and understand why the cosine was used in that situation. Chi et al. found that successful students attempt to explain to themselves how each line in a physics problem is derived from the previous line. These learners also spent less time referring back to examples during problem solving and were more likely to know when they had incomplete or faulty knowledge than poor learners. VanLehn, Jones, & Chi (1991) have investigated this hypothesis by building a computer simulation called Cascade, which models the self{explanation eect. In their model, not all of the procedural knowledge needed to acquire a skill has to be gained while solving problems. Their simulation suggests that many of the rules necessary to solve later problems are acquired solely by studying examples, though extra practice strengthens and tunes those rules once they have been created. In both theories, examples appear to play a major role in learning a new skill (Anderson et al., 1984; Pirolli, 1991; Sweller & Cooper, 1985; VanLehn et al., 1992). In Example Generalization models, the knowledge to solve future problems can be gained simply by studying (and self{explaining) examples (Cooper & Sweller, 1987; Sweller & Cooper, 1985; VanLehn et al., 1992).

7 STUDYING EXAMPLES AND SOLVING PROBLEMS 7 In Knowledge Compilation models, however, the process of studying examples itself can only result in declarative knowledge, and in order to use this example knowledge to acquire domain specic skill, problems must be solved. (Anderson, 1987; Pirolli, 1991). The present series of experiments examines the contributions of studying examples and problem solving to the acquisition of problem solving skill. Specically, these experiments will examine whether it is possible to learn problem solving knowledge by studying examples, or whether problem solving is a necessary aspect of acquiring the skill. Together, these two experiments describe why studying examples is an eective manner of acquiring a skill and, more importantly, describe the situations when students should study examples and the situations when they should solve problems. Experiment 1 If the benet of studying examples derives at least in part from applying an example to help solve a later problem, then this benet should be reduced if the learner's ability to access information from the example is hampered. In all of Sweller's experiments demonstrating the eectiveness of examples for learning, source examples were immediately followed by relevant target problems drawing on the knowledge exemplied in the prior example. Interleaving the examples and problems in this manner may have allowed participants to remember the example and then use that example to guide later problem solving. In contrast, if the source examples were separated from later relevant target problems, then participants would have to select which prior example is applicable, and may have to rely on only partially complete memories of the examples. In knowledge compilation models, learning from examples requires using them to guide later problem solving, hence separating target problems from relevant examples should interfere with learning. On the other hand, if students acquire sucient problem solving skill by studying examples, as argued by example generalization models (e.g., Sweller & Cooper, 1985), then separating the target problem from the source example should not aect later problem solving eectiveness or learning outcomes. The present experiment examines these hypotheses. Method Design We manipulated the order of problems (Interleaved or Blocked sources and targets) and the activity participants performed with the sources (study examples or solve problems). This

8 STUDYING EXAMPLES AND SOLVING PROBLEMS 8 yielded four conditions: Interleaved Example, Interleaved Solve, Blocked Example, and Blocked Solve. Participants could not access previously seen examples or problems. Participants in the two interleaved conditions saw pairs of problems: a source problem (1a, 2a, etc.), immediately followed by the related target problem (1b, 2b, etc.). Participants in the two blocked conditions were given all the sources followed by all the targets (see Table 1). Sources (a's) were similar but not identical to the targets (b's), and each pair of problems was conceptually dierent from every other pair. For example, problems 1a and 1b were similar to each other but dierent from problems 2a and 2b. Participants in the two \example" conditions studied annotated examples as their source problems, while participants in the two \solve" conditions solved their source problems. Participants in each condition solved the same target problems. Participants studied examples and solved problems in the domain of LISP programming. Interleaved Example Example 1a Solve 1b Example 2a Solve 2b Interleaved Solve Solve 1a Solve 1b Solve 2a Solve 2b Blocked Example Example 1a Example 2a Example 3a Example 4a Blocked Solve Solve 1a Solve 2a Solve 3a Solve 4a. Example 4a Solve 4b Example 5a Solve 5b. Solve 4a Solve 4b Solve 5a Solve 5b. Solve 1b Solve 2b Solve 3b Solve 4b. Solve 1b Solve 2b Solve 3b Solve 4b.... Table 1: Tasks for each condition. A's are sources and B's are targets. Identical numbers are similar to each other. Participants The participants were 40 undergraduate paid volunteers from Princeton University and other nearby colleges. 3 Ten participants were assigned to each condition, approximately balanced for Math SAT, an eective predictor of programming ability (Mayer, Dyck, & Vilberg, 1986). Participant Math SATs ranged from 630 to 800, with an overall average of 718. Interleaved Example participants had a 712 average math SAT; Interleaved Solve participants had a 719 average math SAT; Blocked Example participants had a 722 average math SAT; and Blocked Solve participants 3 Data from 3 potential participants were not used because of computer crashes and data from 1 potential participant were not used because the participant took over 2 standard deviations longer than the next slowest participant.

9 STUDYING EXAMPLES AND SOLVING PROBLEMS 9 had a 720 average math SAT. All participants had taken no more than one semester of computer programming and had no prior knowledge of LISP. Apparatus and Materials We constructed source and target problems in a LISP programming curriculum. Source and target solutions were similar but not identical. Cover stories were also constructed to be as dissimilar as possible. This was done in order to discourage analogical mappings from being made on the basis of the cover story (cf. Gentner, 1983; Ross, 1987). A professor wants to know the best and the worst student in each of the discussion sections of his class. He has the names of the participants of each discussion group in lists that start with the best student and end with the worst. Write a function that will take each of these lists { e.g., (smith jones samuels blake) { and return another list containing only the names of the best and the worst student. Here is a solution that denes a function best-worst: => (defun best-worst (students) (cons (first students) (last students))) This function takes one argument which is a list of student names. We can use first to get the rst student's name and last to get a list containing the last student's name. Now we choose the function cons to put these two lists together because the rst student's name is an atom and can simply be inserted into the list containing the last student's name. This will give us our single list of two names. Then we can call the function best-worst as follows: => (best-worst '(smith jones samuels blake)) (smith blake) Table 2: Example of a problem description and annotated source example Annotated source examples were constructed by presenting a typical solution, a brief description of why the code works, and an example of the output that the code produces when run with a particular input. See Table 2 for a sample source example. Participants worked through two chapters of an introductory LISP textbook (Anderson, Corbett, & Reiser, 1987) containing approximately 26 total pages of material using BATBook, an electronic book and problem solving environment (Faries & Reiser, 1988). BATBook (which stands for Behavioral Analogy Tracing Book and Problem Solving Environment) was used because it allowed us to present expository text, examples, problems to solve, and provide an interactive problem solving environment, while recording all interactions, including time spent reading each

10 STUDYING EXAMPLES AND SOLVING PROBLEMS 10 page, searching, studying examples, and problem solving attempts. Students progressed forward and backward through the text by pages and searched the textbook for any word or set of words they could specify. While studying worked out examples or working problems, students were free to search the expository text. A sample BATBook screen is shown in Figure 1. A more complete description of BATBook is presented in Faries & Reiser (1989). Participants worked on assigned problems in BATBook's LISP window (consisting of a simple editor and LISP interpreter). There were six sources and six targets; participants in the two solve conditions solved twelve problems and saw no examples, participants in the two example conditions studied six examples and solved six problems. Participants could test their programs on their own data and submit answers they considered correct. BATBook accepted correct solutions, or briey pointed out data for which the program produced an error or incorrect result. Participants could give up after three incorrect attempts and see a correct answer. While studying worked out examples or solving problems, participants were free to search the expository text. However, they did not have access to prior examples or their prior solutions at any time. Evaluation Questionnaire. Participants were given an evaluation questionnaire when they completed the acquisition phase, to assess their attitude toward their performance and the domain (Reiser, Copen, Ranney, Hamid, & Kimberg, in press). Posttest. A posttest consisting of three near transfer problems followed the evaluation questionnaire. The three problems were near transfer because they used structures and algorithms that were similar to those seen during the acquisition phase, but dierent enough to make them challenging. Participants were free to test their programs in the LISP window, but unlike the learning session, they received no feedback when they submitted answers to the posttest problems. Procedure Participants were given a brief demonstration during Chapter 1 to familiarize them with the learning environment, including reading and searching the text, studying examples (for the two example conditions), and solving problems. Participants then studied examples, solved problems, and read the remaining text at their own pace. Chapter 1 was used to familiarize participants with the BATBook environment, so all participants were given the same sequence of study examples and problems to solve. Chapter 1 introduced students to LISP and some very basic functions (+,?, =,, rst, and rest), and showed students how to use these functions in the LISP interpreter. Chapter 2 introduced students to more complex functions (cons, list, append, reverse, and last)

11 STUDYING EXAMPLES AND SOLVING PROBLEMS 11 Figure 1. Sample screen of BATBook. The textbook and instructions for the student are displayed in the left window. Problems to solve appear in the upper right window. The LISP workspace is the lower right window. When displaying annotated examples, they are displayed in place of the active workspace.

12 STUDYING EXAMPLES AND SOLVING PROBLEMS 12 and showed students how to construct their own functions. Chapter 2 implemented the learning conditions shown in Table 1. Participants took between two and ve total hours in one session to complete the entire learning session and posttest. Typically, the rst thing a participant would do to start a problem in the experimental session was to write a rst program, or defun. The student would then test their program on data of their own choosing or data supplied to them in the problem statement. If running the program returned what they thought it should, they submitted it. If, however, their program did not return what they believed it should return, they were able to use BATBook's simple editor to make modications to the program. Participants then tested and modied their program until they believed it was correct. After participants thought their program was correct, they would submit it. After a program was submitted, BATBook tested the program. If the student's program was correct, the student was able to go to the next problem. If, however, the program was wrong, BATBook described the data that the program would not work on or the error message returned if BATBook's data produced an error. Predictions The two central dierences in predictions between the knowledge compilation and example generalization models concern: (a) the eect of separating example sources and targets, and (b) the eect of solving a block of sources rather than studying them as examples. First, will separating example sources from targets to be solved hamper problem solving performance? According to the example generalization view, learning from examples occurs solely while participants study the examples, so the sequence of examples and problems should not aect how much participants learn. Alternatively, if examples facilitate learning not only because of the elaborations participants generate while studying them (e.g., VanLehn et al., 1992), but also when they are forced to draw on their mental representation of the example to guide later problem solving (Anderson & Thompson, 1989; Pirolli, 1991; VanLehn et al., 1992), then separating a source example from similar target problems should hamper learning, because selecting and remembering an example may be less successful. The comparison of the Interleaved Example and Blocked Example groups evaluates this hypothesis. A second test of whether the benet of examples depends on their use during problem solving concerns the comparison between the two blocked conditions. In previous studies, example generalization theorists found that studying worked out examples led to better learning outcomes than

13 STUDYING EXAMPLES AND SOLVING PROBLEMS 13 solving the same problems (Cooper & Sweller, 1987; Sweller & Cooper, 1985). Their support for this assertion relied on interspersing source examples with target problems (intended to motivate participants to attend to the examples). If the benet of the examples depends in part on their use during problem solving, then the advantage of studying source examples relative to solving them should be reduced if the targets to solve are not interspersed. Thus, if sources and targets are presented in a blocked format, knowledge compilation theories predict that the additional practice the solve condition receives may outweigh the guiding benet of the examples, leading the Blocked Solve participants to learn more than the Blocked Example participants. In contrast, example generalization theorists predict that since the best learning occurs while studying the examples, the participants who are in the Blocked Example condition should learn more than participants in the Blocked Solve condition. Finally, in the case where the sources should be accessible because they immediately precede targets, the predictions of knowledge compilation theories are less straightforward. In this situation, is it better to study the source example or solve the source as a problem? The source example should help guide and simplify problem solving on the target problem, (Anderson, 1982; Pirolli, 1991) compared to participants who solved a source problem. However, since knowledge compilation theorists believe that the number of problems successfully solved is much more important than how easy problems are to solve (e.g., Anderson et al., 1989, 1993), the knowledge compilation prediction is that participants who solve both source and target problems should learn more than participants who only solve the target problems. The prediction of example generalization theories, however, are clear: Studying source examples should produce better learning outcomes than solving source problem. The comparison of the Interleaved Example and Interleaved Solve conditions enables the evaluation of solving versus studying sources that are then accessed during problem solving of targets. The contrasting predictions are summarized in Table 3. Results and Discussion We examined problem solving performance on the acquisition session. Specically, we examined the time to study or solve source problems, the time required to solve target problems, and the accuracy of rst solutions to each target problem. We also examined the accuracy of the submitted solutions to the posttest problems to measure how much participants learned from the various experimental manipulations. We measured program accuracy by counting the minimum

14 STUDYING EXAMPLES AND SOLVING PROBLEMS 14 Theory Predictions Reason Interleaved Example better Examples must be used during problem Knowledge than Blocked Example solving; blocked examples are less accessible. Compilation Blocked Solve better than Examples are poorly remembered. Blocked Example Blocked Example has less practice. Interleaved Solve better The more problems solved than Interleaved Example the more learned. Interleaved Example equal Learning occurs while studying to Blocked Example examples. Equal number of examples. Example Blocked Example better Learning occurs while studying Generalization than Blocked Solve examples, not solving problems; Interleaved Example better Solving problems motivates than Interleaved Solve subjects to attend to examples. Table 3: Predictions of Knowledge Compilation versus Example Generalization theories. \Better" means nal learning outcome. number of program components to be added, deleted, or replaced to render the program a correct solution. Main eects and interactions were conducted using analysis of covariance (ANCOVA) using math SAT as the covariate. If a signicant interaction was found, cell means were examined using a Fisher protected t{test (Fisher, 1952). Otherwise, we examined cell means using a Tukey post{hoc comparison (Tukey, 1953). Chapter 1. All participants received an identical sequence of examples and problems in Chapter 1. Recall that Chapter 1 came prior to the experimental manipulations, which occurred in Chapter 2. Students spent an average of ten minutes reading text and twelve minutes solving eight problems and studying eight examples. Students solved over 98% of the problems correctly in Chapter 1. Recall that when participants believed they had a correct answer, they submitted their answer so that BATBook could test their program for correctness. Therefore, we examined the number of times participants submitted their programs. As expected, there were no dierences in time spent studying examples and solving problems or number of total solution attempts on Chapter 1 (students made an average of 8.3 solution attempts), all Fs non{signicant. These results suggest that participants were appropriately matched in ability level between conditions, and reached equal levels of prociency on the material prerequisite to the experimental manipulations in Chapter 2. It should come as no surprise that it took participants longer to solve source problems than to study source examples (56.6 vs. 7.8 minutes), F (1; 35) = 54:54; MS e = 445:61; p < :0001, but this study is primarily interested in examining how studying an example benets future problem

15 STUDYING EXAMPLES AND SOLVING PROBLEMS 15 Figure 2. Time spent solving Chapter 2 target problems solving, not how much quicker it is to study an example than solve a problem (cf. Cooper & Sweller, 1987; Sweller & Cooper, 1985). While discussing the results of this experiment, we will focus on comparisons of the target problems between conditions. Does separating source examples from target problems hamper learning? Knowledge compilation theories argue that learning from studying examples requires applying information from the example to the problem to be solved. If, however, the source examples are separated from the target problems, participants may not be able to remember the appropriate examples (and the accompanying elaborations) very well (Ross, 1987), and problem solving and learning on the targets may suer. Thus, knowledge compilation predicted that the participants who solved a target problem immediately after the source example (Interleaved Example) would learn more than participants who studied a block of source examples followed by a block of solving target problems (Blocked Example). Consistent with knowledge compilation theories, participants who solved problems interleaved with examples took substantially less time on the target problems than participants who studied a block of source examples and a block of target problems (see Figures 2 and 3), Tukey test, p < :05, suggesting that Interleaved Example participants were able to utilize the source examples better than the Blocked Example participants. Why were Interleaved Example participants able to solve the target problems faster than the

16 STUDYING EXAMPLES AND SOLVING PROBLEMS 16 Blocked Example participants? Perhaps participants in the Interleaved Example condition were able to better recall the source example and use that knowledge to create a better initial program. However, the accuracy of rst solution attempts did not dier for the Interleaved Example condition and the Blocked Example conditions (85% vs. 78%), Tukey test, p > :10. Also, participants in the Blocked Example condition did not submit their program more times than participants in the Interleaved Example condition (12.3 vs. 9.0), Tukey test, p > :10. However, there was a trend in the data suggesting that participants in the Interleaved Example condition created fewer target functions than participants in the Blocked Example condition (14.9 vs. 24.4), though this dierence is not reliable, Tukey test, p > :10. Interleaved Example participants took less time to solve the target problems than Blocked Example participants. Can this dierence be attributed to participants being more motivated to study the examples in the Interleaved Example than in the Blocked Example condition? We have no direct measure of motivation for studying examples in this study, but participants did spend the same amount of time studying the examples in the two conditions (7.9 vs. 7.7 minutes), F < 1, suggesting that they were equally motivated. To investigate if studying examples in an interleaved fashion improved overall learning, we examined two aspects of posttest performance how long it took participants to solve the three problems and how accurate their nal answers were. Participants spent an equal amount of time solving the three posttest problems (34.8 vs minutes), Tukey test, p > :10. However, the posttest scores (Figure 4) were consistent with the learning session results. Participants in the Interleaved Example condition submitted more accurate solutions than participants receiving blocked examples, interaction F (1; 35) = 9:47; MS e = 220; p < :005, Fisher's protected t{test, p < :005. Therefore, participants who studied examples interleaved with solving similar problems constructed target solutions faster and learned more than participants who studied all their examples in a blocked manner, followed by problem solving. We believe that participants in the Blocked Example condition had diculty remembering the source examples, and so were unable to apply an appropriate example to a problem solving episode. Since their access and use of the example was hampered (i.e., they could not remember the example very well), participants in the Blocked Example condition took more time and learned less than participants in the Interleaved Example condition, who could remember and apply an example to a problem solving episode. In short, participants in the Interleaved Example condition had productive problem solving episodes on the target problems, which allowed them to learn more than Blocked Example participants, as evidenced by their posttest scores.

17 STUDYING EXAMPLES AND SOLVING PROBLEMS 17 Figure 3. Time spent on Chapter 2 sources and targets. Error Bars are S.E.M. Is solving sources better than studying examples if the examples are not accessible during subsequent problem solving? The second test of the knowledge compilation view compares participants who solved a block of source problems (Blocked Solve) to those who studied the same block of examples (Blocked Example). The knowledge compilation theory predicts the Blocked Solve participants to exhibit superior problem solving and learning, since Blocked Example participants may have dif- culty drawing upon the examples to guide later problem solving. The additional opportunities to practice and tune problem solving rules would therefore outweigh the potential facilitating eects of guiding examples. Consistent with this hypothesis, participants in the Blocked Solve condition solved the target problems faster than participants in the Blocked Example condition (see Figure 2), Tukey test, p < :01. However, participants in the Blocked Example condition did not submit their target function more times than participants in the Blocked Solve condition (12.3 vs. 14.8), Tukey test, p > :10. Again, there were no dierences in accuracy of rst solutions between the Blocked Solve condition and the Blocked Example condition (86% vs. 78%), F (1; 35) = 2:05; n:s. Tukey test, p > :10. However there was a trend in the data suggesting that Blocked Example participants constructed more solution attempts than did Blocked Solve participants (24.4 vs functions), though this dierence is not reliable, Tukey test, p > :10. Next, we examined how participants performed on the posttest (see Figure 4). Interestingly,

18 STUDYING EXAMPLES AND SOLVING PROBLEMS 18 Figure 4. Grades on the Posttest. there were no dierences in time on the posttest between the Blocked Example and the Blocked Solve conditions (32.3 vs mins), Tukey test, p > :10. As predicted by knowledge compilation, participants in the Blocked Solve condition performed better on the posttest than participants in the Blocked Example condition, Fisher's protected t{test, p < :05. Thus, the Blocked Solve participants solved the target problems more easily and learned more in the process as evidenced by the posttest scores. As discussed earlier, participants in the Blocked Example condition may have had diculty remembering the source example, so they received less facilitation from the examples on the target problems. In contrast, participants in the Blocked Solve condition had to construct their own solutions to the source problems. The extra practice the Blocked Solve participants received allowed them to solve the target problems more quickly. There is also some suggestion that they oundered less on the target problems than participants in the Blocked Example condition, as shown by the trend toward Blocked Example condition creating more functions than the Blocked Solve condition. We believe that the Blocked Solve condition learned more than the Blocked Example condition because the participants in the Blocked Solve condition received extra practice and participants in the Blocked Example condition did not receive the benet of the guidance of their source examples, leading to more oundering and less productive problem solving on the targets.

19 STUDYING EXAMPLES AND SOLVING PROBLEMS 19 Taken together, the comparisons between Interleaved Example vs. Blocked Example and Blocked Example vs. Blocked Solve provide strong support for the knowledge compilation view and against the extreme form of the example generalization view. These results suggest that source examples derive their benet when participants use an example to facilitate their problem solving. If participants are not able to recall the details of a relevant example, the benet of these elaborations is reduced. Subsequent problem solving appears to be required to derive the full benet of studying examples. Accessible sources: solving versus studying. Recall that the knowledge compilation theory predicts that a source example should facilitate problem solving on the target problem, but that the more problems that are solved, the more learning should occur (as long as the problems are solved correctly). That is, knowledge compilation predicts that participants in the Interleaved Example condition should solve the target problems faster than participants in the Interleaved Solve condition, but, since participants in the Interleaved Solve condition solved more problems, they should score better on the posttest than participants in the Interleaved Example condition. However, the prediction of the example generalization theory is clear: Studying source examples should lead to quicker and more accurate performance on both the target problem and the posttest. Here the results were somewhat puzzling. As Figures 2 and 3 suggest, there was a main eect for source: Participants who studied source examples took somewhat longer to solve target problems than participants who solved source problems, F (1; 35) = 12:39; MS e = 398:57; p = :001. However, the dierence between the Interleaved Example and Interleaved Solve conditions is not reliable (42.9 vs mins), interaction F (1; 35) = 1:47; MS e = 398:57; n:s:, Tukey test, p > :10. The accuracy of the rst solutions for Interleaved Example and Interleaved Solve participants also did not dier (85% vs. 87%), F (1; 35) < 1. However, as Figure 4 suggests, participants in the Interleaved Example condition performed better on the posttest than did participants in the Interleaved Solve condition, Fisher's protected t{test, p < :05. Evidently, the examples helped Interleaved Example participants learn more, but did not enable them to solve the targets more quickly. If participants in the Interleaved Example condition learned more than participants in the Interleaved Solve condition, why were they also not able to solve the target problems faster? Apparently, Interleaved Example participants did not create more functions than Interleaved Solve participants (14.9 vs. 12.9), Tukey test, p > :10, nor did Interleave Example participants submit more function than Interleaved Solve participants (9.2 vs. 9.0), Tukey test, p > :10. It appears

20 STUDYING EXAMPLES AND SOLVING PROBLEMS 20 then, that Interleaved Example participants were not writing more functions or trying more dierent ways of solving the same problem than Interleaved Solve participants; rather, it simply took the Interleaved Example participants longer to construct their programs than it did for the Interleaved Solve participants to construct their programs. We also investigated qualitative dierences in target problems between the Interleaved Example and Interleaved Solve conditions by examining the type and number of program changes in terms of syntax and semantics. Each time a function was modied, it was coded in terms of changing syntax (e.g., quotes, parentheses, or mis{spellings) and semantics (e.g., component functions, variables, or data) of a function. A change consisted of adding, deleting, switching places, or replacing the item in question. Note that if a solution had 3 syntactic changes and 2 semantic changes it was coded as a single syntactic change and a single semantic change (i.e., total number of changes were not counted) because in some cases it was dicult to determine absolute numbers of changes (e.g., when the function changed dramatically). An initial comparison between the two conditions of the proportion semantic changes showed that they had non{homogeneous variances, so an arcsin transformation was used. Interestingly, the Interleaved Example condition made proportionally more syntactic changes on targets than the Interleaved Solve condition (63% vs. 38%), F (1; 18) = 4:24; MS e = 0:26; p = :05. Evidently, participants in the Interleaved Example condition focused a large part of their energy while solving target problems on LISP syntax, a very time consuming aspect for novices in learning to program (Miller, Pane, Meter, & Vorthmann, 1994; Soloway, Spohrer, & Littman, 1988). This analysis highlights the dierences between what we will call the instrumental aspects of a task and the more important semantic and algorithmic aspects of a task. The instrumental aspects of learning LISP in this study include the syntax of the domain and how to use the BATBook interface. Learning how to use the interface includes learning how to search the textbook, use the editor, test programs, understand BATBook's error messages, etc. Learning the syntax of LISP includes learning when and where to use quotes and parentheses, how to include functions, variables, and data in programs or function calls, etc. These instrumental tasks are typically very dicult for novices (often because they appear arbitrary), but they are important for a novice to learn because without them students can not solve problems in the domain. However, instrumental aspects of a task are usually viewed as less important than learning the semantics and algorithms of a task. The semantics and algorithms of learning LISP include how each function works, how functions work together, and how dierent algorithms perform dierent tasks. Learning LISP includes learning

21 STUDYING EXAMPLES AND SOLVING PROBLEMS 21 the instrumental aspects of the task along with the semantics and the algorithmic aspects of the task. This analysis examining the syntactic and semantic changes of functions suggests that participants in the Interleaved Solve condition had more practice than participants in the Interleaved Example condition with the instrumental aspects of the task specically the syntax of LISP and the BATBook interface. This dierence is particularly interesting given the other comparisons between the Interleaved Example and Solve conditions. Recall that participants in the Interleaved Example condition learned more than participants in the Interleaved Solve condition as measured by the posttest. However, we found no facilitation on the target problems for the Interleaved Example participants over the Interleaved Solve participants Interleaved Solve participants spent almost 15 minutes less on the target problems than did Interleaved Example participants, and there were no other measures which showed any facilitation for the Interleaved Example condition. The absence of facilitation occurred because the extra practice that participants in the Interleaved Solve condition received allowed them to learn the instrumental aspects of the task more quickly than the Interleaved Example participants. This extra practice the Interleaved Solve participants received on the instrumental aspects of the task overwhelmed any guidance that may have been observed by the Interleaved Example participants. However, the results of the posttest suggest that participants in the Interleaved Example condition were able to learn the semantics and algorithms of LISP better than participants in the Interleaved Solve condition. We believe that studying source examples allowed these participants to learn the semantic and algorithmic aspects of LISP from studying the example and build more eective problem solving structures. Since the posttest tested the semantics and algorithms of LISP, these general (semantic) rules allowed the participants in the Interleaved Example condition to perform better than participants in the Interleaved Solve condition. Learners who studied source examples interleaved with problems also could have been learning the subgoal structure of the problems, as suggested by Catrambone (Catrambone, 1995a, 1995b, 1996, in press). Catrambone has suggested that one of the main advantages to studying examples is that it teaches learners about the subgoal structure of problems. We did not point out the subgoal structure to learners like Catrambone has done (e.g., Catrambone, 1995a), but we feel that students in this experiment could have been learning the appropriate subgoals by studying the examples. Unfortunately, we have no direct evidence of this in our study, so we will not discuss it further. Finally, we evaluated the motivation questionnaire. The thirteen questions were grouped into