Cognitive Psychology and Educational Assessment Robert J. Mislevy 1230-C Benjamin rmislevy@umd.edu Summary This course explores the implications of recent developments in cognitive psychology for educational assessment. While it is not a psychology course per se, we will survey key topics such as expertise research, knowledge representations, information-processing psychology, and the sociocultural perspective on learning. The focus will be on how these ideas are used in designing and using assessments. We will use evidence-centered assessment design to frame our work as we analyze a range of examples, including performance assessment, portfolio assessments, simulation-based tasks, and intelligent tutoring systems. The class meets once a week. The first part of the class will usually be a presentation by the instructor, followed by discussion of the readings and of examples from readings or provided by students. Each student will be responsible for summarizing one of the readings for the class sometime during the semester, and initiating the discussion of that reading. Assignments will also include a series of commentaries or exercises, approximately two of every three weeks, based on the readings and examples. The final project will be the analysis of an assessment of the student s choice. Prerequisites: EDMS 623 (intermediate-level educational measurement: classical test theory, reliability, validity, etc.) and EDMS 651(Intermediate-level statistics: regression, anova), or instructor permission.
Topics and Assignments Class Date Topic Readings Assignment (Due Before Class) 1 1/31 Introduction to class / KWSK, Chap. 1-3 Introduction to cognitive psychology 2 2/7 Evidence-centered assessment design Assessment as argument CPEM, Sections 1&2; Mislevy et al Leverage points ; Greeno, Collins, & 1 paragraph about you Resnick (1997) 3 2/14 Theory-based task design 1 Risconcente & Mislevy; Newstead et al (2002); Hively, Patterson, & Page (1968); *Bao & Redish (n.d.); *TGTE 4 2/21 Design patterns for assessment PADI Research Group; Baxter, Elder, & Glaser (1996) 5 2/28 Expertise Research Ericcson & Smith (1991) Chap 1; Katz (1994); *Mislevy, Steinberg, Breyer, Almond, & Johnson (1999) ; *Camerer & Johnson (1991). 6 3/7 Trait and behavioral perspectives 7 3/14 The information processing perspective Carroll (1976); Hively, Patterson, & Page (1968); *Glaser (1963); *Whitely (1976). VanLehn (1990); Salthouse (1991); *Holyoak (1991); *Simon (1975); *Greeno(1976) 3/21 Spring Break no class 8 3/28 Cognitive diagnosis Embretson (1998); Hendrickson & Mislevy (in press); *Frederiksen & White (1988); *TGTE 9 4/4 Schemas Rumelhart (1980); *Singley & Bennett (2002); *Gentner & Gentner (1983); *TGTE. Questions on evidence-centered design Design pattern based on Katz (1994) Commentary for Carroll (1976) Commentary for VanLehn (1990) Commentary for Embretson (1998)
4/11 AERA no class 10 4/18 The sociocultural/situative perspective 11 4/25 Info processing vs sociocultural debate Brown, Collins, & Duguid (1989); Saxe (1988); Gipps (1999); *Resnick (1997); *TGTE. Anderson, Reder, & Simon (1996 & 1997); Greeno (1997); Anderson, Greeno, Reder, & Simon (2000). 12 5/2 Intelligent tutoring systems Mislevy & Gitomer (1996); *Steinberg & Gitomer (1996); *Corbett & Anderson (1992) 13 5/9 Theory-based task design 2 *Kindfield (1999); *Bejar (1993);*TGTE 5/16 Finals week Project due 5/20 Design pattern based on Embretson (1998) Prospectus for final project Final project: Analysis + (design pattern and/or Toulmin diagram) Boldface indicates focus reading for class discussion * indicates a paper that a student will summarize for the class. CPEM = Cognitive psychology and educational measurement, Mislevy (in preparation) KWSK = Knowing what students know, National Research Council (2002); Pellegrino, Chudowsky, & Glaser (Eds.) *TGTE = A class of expertise chapter from Toward a general theory of expertise, Ericcson & Smith (Eds.) (1991): Allard & Starkes on motor skills, Anzai on physics, Charness on chess, Dorner & Scholkopf on controlling complex systems, Patel & Groen on medical expertise, Scardamalia & Bereiter on literate expertise, Sloboda on musical expertise.
Textbooks Ericcson, K.A., & Smith, J. (Eds.) (1991). Toward a general theory of expertise. Cambridge, England: Cambridge University Press. National Research Council (2002). Knowing what students know. J. Pellegrino, N. Chudowsky, & R. Glaser (Eds.). Washington, D.C.: National Academy Press. Common Readings Anderson, J.R., Greeno, J.G., Reder, L.M., & Simon, H.A. (2000). Perspectives on learning, thinking, and activity. Educational Researcher, 29, 11-13. Anderson, J.R., Reder, L.M., & Simon, H.A. (1996). Situated learning and education. Educational Researcher, 25, 5-11. Anderson, J.R., Reder, L.M., & Simon, H.A. (1997). Situative versus cognitive perspectives: Form versus substance. Educational Researcher, 26, 18-21. Baxter, G. P., Elder, A. D., & Glaser, R., (1996). Knowledge-based cognition and performance assessment in the science classroom. Educational Psychologist, 31 (2), 133-140. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42. Carroll, J.B. (1976). Psychometric tests as cognitive tasks: A new structure of intellect. In L.B. Resnick (Ed.), The nature of intelligence (pp. 27-56). Hillsdale, NJ: Erlbaum. Embretson, S.E. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3, 380-396. *Ericcson, K.A., & Smith, J. (1991). Prospects and limits of the empirical study of expertise: An introduction. In K.A.Ericcson & J. Smith (Eds.), Toward a general theory of expertise (1-38). Cambridge, England: Cambridge University Press. Gipps, C. (1999). Socio -cultural aspects of assessment. Review of Research in Education, 24, 355-392. Greeno, J.G. (1997). On claims that answer the wrong questions. Educational Researcher, 26, 5-17. Greeno, J.G., Collins, A.M., & Resnick, L.B. (1997). Cognition and learning. In D. Berliner & R. Calfee (Eds.), Handbook of educational psychology (pp. 15-47). New York: Simon & Schuster Macmillan. **Hendrickson, A., & Mislevy, R.J. (in press). Cognitively Based IRT Models. In B. Everitt & D. Howell (Eds.), Encyclopedia of Statistics in Behavioral Science. New York: Josset-Bass/Wiley Hively, W., Patterson, H.L., & Page, S.H. (1968). A "universe-defined" system of arithmetic achievement tests. Journal of Educational Measurement, 5, 275-290. Katz, I.R. (1994). Coping with the complexity of design: Avoiding conflicts and prioritizing constraints. In A. Ram, N. Nersessian, & M. Recker (Eds.), Proceedings of the Sixteenth Annual Meeting of the Cognitive Science Society (pp.485-489). Hillsdale, NJ: Erlbaum. **Mislevy, R.J. (in preparation). Cognitive psychology and educational measurement.
**Mislevy, R.J., & Gitomer, D.H. (1996). The role of probability-based inference in an intelligent tutoring system. User-Modeling and User-Adapted Interaction, 5, 253-282. Available online as CSE Technical Report 413. Los Angeles: The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA. **Mislevy, R.J., & Risconscente, M. (in press). Evidence-Centered Assessment Design: Layers, Structures, and Terminology. To appear in S. Downing & T. Haladyna (Eds.), Handbook on Test Development. Mahwah, NJ: Erlbaum **Mislevy, R.J., Steinberg, L.S., Almond, R.G., Haertel, G.D., & Penuel, W.R. (in press). Leverage points for improving educational assessment. In B. Means & G.D. Haertel, Designs for evaluating the effects of technology in education. Newstead, S., Bradon, P., Handley, S., Evans, J., & Dennis, I. (2002). Using the psychology of reasoning to predict the difficulty of analytical reasoning problems. In S.H. Irvine & P.C. Kyllonen (Eds.), Item generation for test development (pp. 35-52). Mahwah, NJ: Erlbaum. **PADI Research Group (2004). Design patterns for assessing science inquiry. Menlo Park, CA: SRI, International. Rumelhart, D.A. (1980). Schemata: The building blocks of cognition. In R. Spiro, B. Bruce, & W. Brewer (Eds.), Theoretical issues in reading comprehension (pp. 33-58). Hillsdale, NJ: Erlbaum. *Salthouse, T.A. (1991). Expertise as the circumvention of human processing limitations. In K.A.Ericcson & J. Smith (Eds.), Toward a general theory of expertise (286-300). Cambridge, England: Cambridge University Press. Saxe, G.B. (1988). Candy selling and math learning. Educational Researcher, 17(6), 14-21. VanLehn, K. (1990). Problem-solving and cognitive skill acquisition. In M. Posner (Ed.), The foundations of cognitive science (pp. 527-580). Cambridge, MA: MIT Press. * In textbook ** Will be provided on line Individual Readings (A student summarizes for the class) *Allard, F., & Starkes, J.L. (1991). Motor-skill experts in sports, dance, and other domains. In K.A.Ericcson & J. Smith (Eds.), Toward a general theory of expertise (126-152). Cambridge, England: Cambridge University Press. *Anzai, Y. (1991). Learning and use of representations for physics expertise. In K.A.Ericcson & J. Smith (Eds.), Toward a general theory of expertise (64-92). Cambridge, England: Cambridge University Press. Bao, L., & Redish, E.F. (n.d.). Model analysis: Assessing the dynamics of student learning. Unpublished manuscript.
Bejar, I.I. (1993). A generative approach to psychological and educational measurement. In N. Frederiksen, R.J. Mislevy, & I.I. Bejar (Eds.), Test theory for a new generation of tests (pp. 323-357). Hillsdale, NJ: Erlbaum. *Camerer, C.F., & Johnson, E.J. (1991). The process-performance paradox in expert judgment: How can experts know so much and predict so badly? In K.A. Ericcson & J. Smith (Eds.), Toward a general theory of expertise (195-217). Cambridge, England: Cambridge University Press. *Charness, N. (1991). Expertise in chess: The balance between knowledge and search. In K.A.Ericcson & J. Smith (Eds.), Toward a general theory of expertise (38-63). Cambridge, England: Cambridge University Press. Corbett, A.T. and Anderson, J.R. (1992). Knowledge tracing in the ACT Programming Tutor. The Proceedings of the Fourteenth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Erlbaum. *Dorner, D., & Scholkopf, J. (1991). Controlling complex systems; or, Expertise as grandmother s know-how. In K.A.Ericcson & J. Smith (Eds.), Toward a general theory of expertise (218-239). Cambridge, England: Cambridge University Press. Frederiksen, J.R., & White, B.Y. (1988). Implicit testing within an intelligent tutoring system. Machine-Mediated Learning, 2, 351-372. Gentner, D., & Gentner, D.R. (1983). Flowing waters or teeming crowds: Mental models of electricity. In D. Gentner & A.L. Stevens (Eds.), Mental models (pp. 99-129). Hillsdale, NJ: Erlbaum. Gitomer, D.H., Steinberg, L.S., & Mislevy, R.J. (1995). Diagnostic assessment of trouble-shooting skill in an intelligent tutoring system. In P. Nichols, S. Chipman, & R. Brennan (Eds.), Cognitively diagnostic assessment (pp. 73-101). Hillsdale, NJ: Erlbaum. Glaser, R. (1963). Instructional technology and the measurement of learning outcomes: Some questions. American Psychologist, 118, 519-521. Greeno, J.G. (1976). Cognitive objectives of instruction: Theory of knowledge for solving problems and answering questions. In D. Klahr (Ed.), Cognition and instruction (pp. 123-159). Hillsdale, NJ: Erlbaum. Kindfield, A.C.H. (1999). Generating and using diagrams to learn and reason about biological processes. Journal of Structural Learning and Intelligent Systems, 18, 81-124. Mislevy, R.J., Steinberg, L.S., Breyer, F.J., Almond, R.G., & Johnson, L. (1999). A cognitive task analysis, with implications for designing a simulation-based assessment system. Computers and Human Behavior, 15, 335-374. [CRESST RR--http://www.cse.ucla.edu/CRESST/Reports/TECH487.pdf] *Patel, V.L., & Groen, G.J. (1991). The general and specific nature of medical expertise: A critical look. In K.A.Ericcson & J. Smith (Eds.), Toward a general theory of expertise (93-125). Cambridge, England: Cambridge University Press. Resnick, L.B. (1997). Student performance portfolios. In H.J.Walberg & G.D. Haertel(Eds.), Psychology and educational practice (pp. 158-175). Berkeley: McCutchan. *Scardamalia, M., & Bereiter. (1991). Literate expertise. In K.A.Ericcson & J. Smith (Eds.), Toward a general theory of expertise (172-194). Cambridge, England: Cambridge University Press. Simon, H. (1975). The functional equivalence of problem solving skills. Cognitive Psychology, 7, 268-288.
Singley, K., & Bennett, R.E. (2002). Item generation and beyond: Applications of schema theory to mathematics assessment. In S.H. Irvine & P.C. Kyllonen (Eds.), Item generation for test development (pp. 361-384). Mahwah, NJ: Erlbaum. *Sloboda, J. (1991). Musical expertise. In K.A.Ericcson & J. Smith (Eds.), Toward a general theory of expertise (153-171). Cambridge, England: Cambridge University Press. Steinberg, L.S., & Gitomer, D.G. (1996). Intelligent tutoring and assessment built on an understanding of a technical problem-solving task. Instructional Science, 24, 223-258. Whitely, S.E. (1976). Solving verbal analogies: Some cognitive components of intelligence test items. Journal of Educational Psychology, 68, 234-242. * In textbook ** Will be provided on line