The Acquisition of Expertise in Software Engineering Education

The Acquisition of Expertise in Software Engineering Education Judith E. Sims-Knight Psychology Department jsimsknight@umassd.edu Abstract University of Massachusetts Dartmouth N. Dartmouth, MA 02747-2300 Our goal in this paper is to examine the nature of expertise in software development and then to analyze the components of process education necessary to promote its development. This analysis includes three aspects of expertise: (a) declarative knowledge ( knowledge that ), (b) procedural knowledge ( how to knowledge ), and (c) metacognitive knowledge (self-monitoring, agency, reflection). These flavors of knowing play an integral role in software development. Traditional education with its emphasis on textbooks and lectures as its primary pedagogical tools can teach declarative knowledge, but that knowledge is not automatically connected with doing. The major impact of process education has been to provide a more effective learning environment that focuses on the development of procedural knowledge. It has become clear that procedural and declarative knowledge alone are not sufficient for the development of expertise. Students need to learn how to reflect on both the problem at hand and on the ways they are solving the problem if they are to become agents for their own improvement. Although process educators are often aware and desirous of facilitating metacognitive knowledge, it is rarely addressed explicitly. In this paper we connect issues of metacognitive knowledge to the evolution of software development skill. Our analysis leads to recommendations for structuring software engineering education activities throughout the students program of study to maximize the development of all three components of expertise. We include examples from our work on integrating process activities in undergraduate computer science instruction. I. Introduction We have claimed elsewhere [28][29][30] that software process concepts and practices should be integrated into software engineering curricula. Furthermore, we have contended that those software process activities that are incorporated into the program should be modeled after practices identified as essential to support good software development efforts, referred to as best practices [7]. Incorporating best practices into curricula may have value in and of itself. Exposing students to a more authentic context for learning software development appeals to our understanding and expectations of the learning experience. Richard L. Upchurch CIS Department rupchurch@umassd.edu But mere exposure does not guarantee that students will develop the skill necessary to apply these practices effectively. Furthermore, it does not guarantee that these practices will persist past the current academic term. Research on the nature of expertise makes it very clear that expertise is a very complex, multifaceted aggregation of skills that develop only when practiced explicitly by the individual, whether rank novice or expert. Thus, to construct effective educational situations, software educators must understand the nature of expertise in the software engineering context and how to structure learning environments to encourage the development of these cognitive skills [22]. In this paper we will first review the evidence on the nature of expertise as it relates to software engineering, and then we will show how and why the process approach can promote the development of expertise. Then we will describe how we have incorporated these principles into a sophomore level computer science course. II. Why Do We Need to Understand Expertise? Experts are often characterized as having three distinctive kinds of knowledge: (a) declarative knowledge ( knowledge that ), (b) procedural knowledge ( how to knowledge ), and (c) metacognitive knowledge (self-monitoring, agency, reflection). Declarative knowledge refers to the kind of knowledge typically learned from textbooks--facts and concepts. Procedural knowledge refers to being able to do something, be it writing code, proceeding through analysis and design, using a software process approach, or writing a paper about ethics in the software industry. Metacognitive knowledge refers to a person s skill at planning strategy, monitoring process and progress, changing what one is doing when appropriate, and reflecting on the process so that one can discover ways to improve. Students typically learn by reading textbooks, listening to lectures, and taking exams on the material so learned; this kind of learning is declarative. In traditional programming courses this strategy is augmented by programming assignments; this requires procedural learning. When programming assignments are merely added as homework assignments, the implicit assumptions is that the declarative knowledge is a sufficient basis for procedural learning to occur. The acquisition of

metacognitive skills is rarely addressed at all, either by instruction or by assignment. Research on expertise, both within the software domain and in other fields, makes it very clear each of these components must be learned explicitly. For example, studying instructional text is not a sufficient basis for students to solve LISP programs, whereas doing one programming problem improves the probability of doing a second one by 50% [3]; adding an instructional example of how to construct the program produce improvements of over 60% [24]. The same is true in learning to use computers [27]. In fact, the widely used approach to instruction called the cognitive apprenticeship model [11] is predicated on the assumption that procedural knowledge needs to be explicitly addressed in instruction. It is equally true that one can not teach procedural skill by teaching procedural principles. For example, Ratcliff and Siddiqi [26] found that teaching students the principles of top-down design was not sufficient to enable them to practice top-down design. Although less studied, a similar claim can be made for metacognitive knowledge. When solving problems, most university students do not spontaneously consider strategies, plan their approach, evaluate their progress, and think through how to change what they are doing. When asked to think metacognitively at every step of a problem-solving episode, such students can do so and through doing so develop deeper understanding and better performance on subsequent problems [6]. These representative studies demonstrate that learning declarative knowledge from textbooks and lectures addresses only part of what is needed to develop expertise. To devise effective instructional techniques, it is necessary to understand both the nature of expertise and how it is acquired. III. The Nature of Expertise How do experts differ from novices? Observational studies of experts in a variety of areas (e.g., programming, software design, medical diagnosis, chess, musical performance, sports) have shown a common set of problem-solving skills. Experts can solve routine problems quickly, efficiently, and with little wasted effort. In fact solving familiar problems becomes so automatic that experts find it difficult to explain how they do so to novices (which impairs experts abilities to teach problem solving). Second, when faced with unusual problems experts can quickly accommodate the new features. Third, experts are more likely to recognize when their business-as-usual solutions won t work and are able to discard familiar solutions and build new ones [14]. When faced with novel situations their metacognitive skills permit them to work their way through the problem more effectively because they monitor their progress better, recognize earlier when a potential solution is promising and when it is not viable, and move quickly to a new direction. This description is exemplified in the research on expert software designers. Experts typically use top-down, breadthfirst strategies with familiar problems. When they encounter some unusual aspects, they switch to a depth-first approach to explore the parameters of the unusual feature. Once they understand how to solve that particular feature they return to their top-down strategy. Most instances of what is typically called opportunistic design fits into this pattern of occasional forays into depth-first exploration of unusual features [4]. It is likely that in those instances where experts exhibit true opportunistic design, they are working beyond their domain-specific knowledge and are exploring how to create a solution afresh. What is it about experts that makes this problemsolving expertise possible? The primary difference between experts and novices is NOT in basic intellectual processes, such as IQ, memory, or speed of processing. Rather, the distinguishing feature of expertise is that experts have extraordinary action-based knowledge in their field. For example, chess players have extraordinary memories for meaningful chessboard configurations, but do not differ from novices in their memory for random chessboard configurations. The meaningful chessboard configurations are conceptualized in terms of their playability, that is, whether they will lead to a checkmate. In physics experts have been shown to represent problems in terms of relatively abstract principles, such as energy conservation and Newton s second law whereas novices represent problems in terms of their surface features, such as springs or inclined planes [9] (see [13] and [33] for discussions of these issues). Results from research on computer programming is consistent with this picture expert programmers conceptualize programs in terms of what the program needs to do, whereas novices conceptualize programs in terms of how the program operates [1][2][23]. The more abstract representation used by the expert permits designers to get beyond the surface features of a program to a more meaningful level. IV. How Expertise Develops Research on the acquisition of expertise has shown that (a) it takes a long time, (b) it takes intense, focused effort, and (c) it can not be taught by direct tuition. One of the most widely replicated findings in all domains studied is that it takes at least 10 years to reach expert level. To reach grandmaster chess level, for example, is estimated to take 32,000 hours of practice [8]. Ten years of exposure to a field is a necessary, but not sufficient condition for expertise. The number of years of experience is a weak predictor of performance. For instance,

Vessey [32] found number of years of experience was not a good predictor in debugging skill. Practice alone is sufficient to develop a certain level of competence, but, for most people, will result in an asymptotic level below that of true excellence. To develop beyond this asymptotic level requires something more. Ericsson, Krampe, and Tesch- Römer [13] describe that something more as deliberate practice, which they define as practice in order to improve. Ericsson et al. found that the major difference between music conservatory students who were rated by the faculty as being future solo and symphony players versus other students was the total amount of time they had spent practicing alone. The number of hours they played their instrument (by themselves or with groups) was not the operative factor. Likewise, the factor that best predicts chess expertise is the amount spent alone studying games [8]. In both cases, the operative variable was not simply the amount of time engaged in the activity (e. g., playing in ensembles or competing in tournaments). The crucial ingredient was practice with the intent of figuring out how to get better. The picture presented so far is that expertise is something that results from working at it. It stands to reason that if students are to work that hard at improvement, they will do so only if they believe that they can and will succeed through their efforts. This has been experimentally demonstrated in a study of organizational decision making with graduate business students. Half the students were told that in acquiring a new skill, people do not begin with faultless performance. However, the more they practice, the more capable they become. The other half were told that the task would identify whether they had the underlying ability to be good managers. The group who expected to improve with practice showed continued improvement with practice, whereas the fixed ability group showed a steady decline in performance goals, efficiency of problem solving, and actual performance [31]. Furthermore, in studies of the effectiveness of instruction in mathematics it has been found that effective instructional techniques have their major impact through improving students tendencies to persist and feelings of self-efficacy [5]. Similarly, increasing students persistence and thereby time-on-task has been found to increase the quality of writing [18]. What is being learned during deliberate practice? Certainly students are learning the domain-specific knowledge base that provides experts with their efficient problem solving of familiar domains, but they are also learning metacognitive skills. Several studies have demonstrated that (a) students who reflect on what they are learning learn better both on declarative and procedural tasks (see [25] for an example in the programming domain), and (b) inducing students to reflect upon the material is effective, thus demonstrating that it is the metacognitive activity that produces the improved performance [6][10]. V. Professional Practice Wholesale adoption of industrial best practices by the software education community is doomed. The research described in this paper demonstrates very clearly that it takes more than a description of principles to be able to apply those principles effectively. To develop robust practices, students must assimilate these practices into their personal development activities. Practices in the software industry have been moving away from mere presentations of laundry lists to models more consistent with the research on expertise. In particular, the quality paradigm, as exemplified by the Capability Maturity Model, explicitly focuses on the process of creating software rather than on the products. To achieve this software developers systematically collect data and use the data to help them decide how to change their behaviors. Thus, it requires that they focus on the processes that produce defects and the development of strategies to reduce errors. In other words,, it encourages software developers to focus on their own metacognitions. In the process approach software developers follow a sequence of steps. They musta) understand how they work now (which includes evaluation based on data); b) conceptualize how they want to work; c) identify process changes; d) plan the improvement activities; e) find the resources needed by the plan; f) execute the plan, and g) repeat the improvement process [21]. The mandate for developers then is to learn the skills necessary to control their work processes as well as to conduct the technical activities involved in software design. VI. How process education promotes the development of expertise Let us summarize the argument so far. First, the long-term goal of education should be to produce effective practitioners, or at least provide the groundwork for that goal. It is clear that universities are failing to reach that goal [20]. Second, to reach this goal it is necessary for educators to attend explicitly to providing learning environments in which their students can develop procedural and metacognitive skills as well as declarative knowledge. Third, the most important aspects of instruction are not the format of the material, but rather providing conditions under which students can develop habits of work that will sustain them through 10 years of deliberate practice. A process orientation is crucial, so that students can see how to develop better strategies and to do this repeatedly. These metacognitive skills in turn will demonstrate to students that through hard work and repeated practice they can improve in ways that can be measured and such experiences should result in increased tendencies toward persistence and in feelings of self-efficacy.

A process focus, supported by measurement, makes students development activity visible to them. By making the process visible, it becomes understandable, accessible, and eventually controllable [16]. We integrate best practices into the development activities of our undergraduates students (the activities reported here are used with sophomore computer science majors). Students learn to develop appropriate process descriptions for how to go about developing software and they learn how to regulate the execution of that script. Students are asked to plan how they will proceed in a project and to document the product as they do the work. After completing an artifact, other students conduct a formal review of the work product using a defined process [30]. In peer reviews, students assume the role of author and reviewers. The author provides the rationale for certain decisions and describes that rationale to the reviewers. The reviewers request clarification and ask questions. The instructor provides scaffolding for the groups, encouraging questions and comments, and interceding as necessary to keep the groups on track. Checklists, used during the preparation and conduct of the reviews, serve two purposes. First, they provide an important model based on an expert s view of how to approach a task and what to look for. Second, they provide scaffolding for thought and action. These reviews include defect identification and difficulty rating. All the information produced is captured and stored electronically in design notebooks available via a standard web browser. Once the review is complete, the students discuss the remarks as a group. Each student has the opportunity to work in each of three identified roles (author, reviewer, recorder) in a review during the course. Experiencing a defined review process for each major artifact in software development (i. e., requirements, design, and code), with roles for each person, allows students to see the effectiveness of this activity as a defect removal strategy. Furthermore, during reviews, students see how others solved problems and must articulate reasons for their design decisions. For each project, all students complete a postmortem. In the postmortem, they document the development process they used for that project and use the size, effort, and defect data from their project, along with self-reflection, to assess their development process, and establish personal, measurable goals for the next project. The results of each postmortem are also stored electronically in their design notebooks. By maintaining records for each project the students have a detailed record of their activity over the course of the semester and can review it throughout the semester. A culminating event for the semester is the student s personal, critical assessment of how their practices changed during the course. VII. Conclusions Incorporating software processes into our curriculum provides many of the necessary conditions for the development of expertise. First, it sets up conditions whereby students can incorporate the declarative knowledge they learn from texts into the real world of design and programming. Second, it provides a framework for practice, not only of design and programming, but also of the software processes as they are practiced in the software industry. These two conditions promote the development of decontextualized representations of declarative/procedural schemas that appear to be essential in operationalizing expertise [33]. These would be sufficient reasons to adopt process education, but it may be that the most important advantage of process education is that in its framework lies the key to helping students develop metacognitive skills (see [17] for a broader perspective on changing agency for learning). First, through expert modeling and practice, with coaching, students learn to create plans for how to complete their products and organize their activities. Second, reviewing the work of others helps students to adopt attitudes of constructive criticism that they can then apply to their own work. Third, through systematic measurement and qualitative postmortems students develop effective skills at self-reflection. The principled focus on the metacognitive skills [12] embedded in these activities appears to be one of the keys to facilitating the evolution of higher levels of competence. They learn to think about how and why they work in particular ways, and how to develop strategies for altering ineffective habits. Finally, it teaches students that they are themselves works in progress who can get better and better at what they do--and more and more efficient if only they work intelligently within the process framework. Finally, students begin recognizing and exercising increasing levels of control over their own work. This degree of empowerment provides the catalyst for working smarter. This results in increased time-on-task accompanied by an increase in the quality of the time-on-task. Our studies, and those being conducted to determine the effectiveness of the Personal Software Process (PSP) [15][19], indicate substantial gains by students. Note: This project is supported, in part, by grant DUE- 9555042 from the National Science Foundation. Material developed under this project is freely available from http://www2.umassd.edu/swpi/nsf/material.html. V. References [1] Adelson, B. Problem solving and the development of abstract categories in programming languages, Memory & Cognition, 9, 1981, pp. 422-433

[2] Adelson, B. When novices surpass experts: The difficulty of a task may increase with expertise, Journal of Experimental Psychology: Learning, Memory, Cognition, 10, 1984, pp. 483-495. [3] Anderson, J. R., Cornrad, F., & Corbett, A. Skill acquisition and the Lisp Tutor, Cognitive Science, 13, 1989, pp. 467-505. [4] Ball, L. J., & Ormerod, T. C. Structured and opportunistic processing in design: A critical discussion, International Journal of Human-Computer Studies, 43, 1995, pp. 131-151. [5] Bandura, A. Self-efficacy: The exercise of control. New York: Freeman, 1997. [6] Berardi-Colletta, B., Buyer, L. S., Dominowski, R. L., & Rellinger, E. R. Metacognition and problem solving: A process-oriented approach, Jounral of Experimental Psychology: Learning, Memory, and Cognition, 21, 1995, pp. 205-221. [7] Brown, N. Industrial-Strength Management Strategies, IEEE Software, July 1996, pp. 94-103. [8] Charness, N., Mayr, U., Krampe, R. The role of practice and coaching in entrepreneurial skill domains: An international comparison of life-span chess skill acquisition, K. A. Ericsson (ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports, and games. Mahwah, NJ: Erlbaum, 1996, pp.51-80. [9] Chi, M. T. H., Feltovich, P., & Glaser, R. Categorization and representation of physics problems by experts and novices, Cognitive Science, 5, 1981, 121-152. [10] Chi, M., de Leeuw, N., Chiu, M., & LaVancher, C. Eliciting self-explanations improves understanding, Cognitive Science, 18, 1994, pp. 439-477. [11] Collins, A., Brown, J. & Newman, S. Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics, in Knowing, learning, and instruction: Essays in honor of Robert Glaser, L. B. Resnick, Editor. Erlbaum :Hillsdale, NJ, 1989, pp. 453-494. [12] Davidson, J. E., Deuser, R. & Sternberg, R. J. The Role of Metacognition in Problem Solving, J. Metcalfe & A. P. Shimamura (eds.), Metacognition: knowing about knowing. Cambridge, MA: MIT Press, 1994, pp. 207-226. [13] Ericsson, K. A., Krampe, R., & Tesch-Römer, C. The Role of Deliberate Practice in the Acquisition of Expert Performance, Psychological Review, 3, 1993, pp. 363-406. [14] Feltovich, P. J., Spiro, R. J., & Coulson, R. L. Issues of expert flexibility in contexts characterized by complexity and change, In Feltovich, P. J., Ford, K. M., & Hoffman, R. R. (Eds.) Expertise in context: Human and machine. Menlo Park, CA/Cambridge, MA: AAAI Press/MIT Press, 1997, pp. 43-65. [15] Ferguson, P., Humphrey, W. S., Khajenoori, S., Macke,, S., & Matvya, A. Results of Apply the Personal Software Process, IEEE Computer, May 1997, pp. 24-31. [16] Fox, C. & Frakes, W. The Quality Approach Is It Delivering? Communications of the ACM, 40, 1997, pp. 25-29. [17] Glaser, R. Changing the agency for learning: Acquisition of expert performance, K. Anders Ericsson (ed.) The road to excellence: The acquisition of expert performance in the arts and sciences, sports, and games. Mahwah, NJ: Erlbaum, 1996, pp. 303-311 [18] Hayes, J. R. & Nash, J. G. On the Nature of Planning in Writing, The Science of Writing: Theories, Methods, Individual Differences, and Applications. Mahway,NJ: Lawrence Erlbaum, 1996, pp. 29-55. [19] Hayes, W. & Over, J. The Personal Software Process (PSP): An Empirical Study of the Impact of PSP on Individual Engineers, CMU/SEI-97-TR-001, 1997. [20] Hilburn, T. Software Engineering Education: A Modest Proposal, IEEE Software, November/December 1997, pp. 44-48. [21] Hsia, PP. Learning to Put Lessons Into Practice, IEEE Software, September 1993, pp. 14-17. [22] Linn, M. C. Designing Computer Learning Environments for Engineering and Computer Science: The Scaffolded Knowledge Integration Framework, Journal of Science Education and Technology, 4, 1995, pp. 103-126. [23] McKeithen, K. B., Reitman, J. S., Rueter, H. H., & Hirtle, S. C. Knowledge organizations and skill differences in computer programmers, Cognitive Psychology, 13, 1981, pp. 307-325. [24] Pirolli, P. Effects of examples and their explanation in a lesson on recursion: A production system analysis, Cognition and Instruction, 8, 1991, pp. 207-259. [25] Pirolli, P. & Recker, M. Learning Strategies and Transfer in the Domain of Programming, Cognition and Instruction, 12, 1994. [26] Ratcliff, B. & Siddiqi, J. An empirical investigation into problem decomposition strategies used in program design, International Journal of Man-Machine Studies, 22, 1984, pp. 77-90. [27] Reder, L. M., Charney, D. H., & Morgan, K. I. The role of elaborations in learning a skill from an instructional text, Memory & Cognition, 14, 64-78. [28] Upchurch, R., & Sims-Knight, J. E. Integrating Software Process in Computer Science Curriculum,. Frontiers in Education Conference, Pittsburgh, PA, November 5-8, 1997.

[29] Upchurch, R., & Sims-Knight, J. E. Designing Process-Based Software Curriculum, Proceedings of the Tenth Conference on Software Education and Training, Virginia Beach, VA, April 13-16, 1997. Los Alamitos: IEEE Computer Society Press, pp. 28-38. [30] Upchurch, R., & Sims-Knight, J. E. In Support of Student Process Improvement, Proceedings of CSEE&T'98, Atlanta, Georgia, February 22-25, 1998. [31] Wood, R. & Bandura, A. Social cognitive theory of organizational management, Special issue: Theory development forum. Academy of Management Review, 14, 1989, pp. 361-384. [32] Vessey, I. Expertise in Debugging Computer Programs: A Process Analysis, International Journal of Man-Machine Studies, 23, 1985, pp. 459-494. [33] Zeitz, C. Some Concrete Advantages of Abstraction: How Experts Representations Facilitate Reasoning, in P. J. Feltovich, K. M. Ford, & R. R. Hoffman (eds.), Expertise in Context. Meno Park, CA: AAAI Press, 1997, pp. 43-65.