Design Strategies to Improve the Validity of Learning, Assessment, and Evaluation

Transcription

1 Design Strategies to Improve the Validity of Learning, Assessment, and Evaluation Eva L. Baker UCLA/CRESST Segundo Congreso Latinoamericano de Medición y Evaluación Educacional (COLMEE) María Isabel Sheraton México DF March 13, 2015 Regents of University of California 1

2 New Ways to Think of Design, Assessment, Evaluation, and Validity in Education Regents of University of California 2

3 Today s Goals How to Improve Learning Using Design Features Assessment with Validity Regents of University of California 3

4 Overview Terms defined Importance of design features in education Its impact on the validity inferences about education and training Examples of learning game designs, teacher value-added evaluation, and large scale student assessment Regents of University of California 4

5 What CRESST Does Standards-based assessment design and validation for formative and summative uses Measurement advances for: Technology-based tests and performance assessments Training, games, and simulations Methodology New approaches to validity, data mining, automated scoring Regents of University of California 5

6 Definitions Design: Systematically making models, artifacts, or programs to achieve goals Assessment: Measuring learning during and following instruction Evaluation: Making judgments of value using evidence Validity: Inferences made from assessment and evaluation warranted given their purpose(s) Regents of University of California 6

7 Design and Validity are Built-in Real quality must be the source of the subjects and objects, the cone from which the tree must start. Zen and the Art of Motorcycle Maintenance --R.M. Pirsig, 1974 Regents of University of California 7

8 Design In educational research and development, design has at least two different meanings: 1. The design of a process or artifact, like a curriculum, lesson, examination, or game 2. The way in which one arranges a research and evaluation study Regents of University of California 8

9 Designing To make something new systematically, one creates an original design or adapts one already in use Designer is a name used to describe many different kinds of people: Design websites, dress celebrities, plan cities, decide on color schemes, or make cars, appliances, strollers, clothes, educational material Regents of University of California 9

10 Architecture is a Design Field Regents of University of California 10

11 Design Design encompasses both about the surface and what is underneath - the design infrastructure Regents of University of California 11

12 Different Ways to Enter Design Problems Start at the beginning Begin at the end and work backward to the start Start in the middle and work both ways Regents of University of California 12

13 Design from the Beginning Statement of intentions, standards, or goals Very general statements for communication Specifications More detail about content, cognition, ontologies sequence, format, and style Sample items or instructional example To illustrate the specifications Regents of University of California 13

14 Design from the End Looking at criterion, job performance, tests, and expected skills in use Work backward to design intermediate and beginning knowledge (required prior knowledge or entry skills) Problems: May too narrowly describe goals May encourage practicing the test rather than learning more than what is measured Regents of University of California 14

15 Goals or Standards Drive Student Learning (or Their Assessments) In the US and elsewhere, these goals have been recently revised Most focus on content but cognition -- practices play a role Common Core K-12 Mathematics and Language Arts Next Generation Science Standards Regents of University of California 15

16 Infrastructure Ontologies: Networks of Relationships SEL Problem Solving Physics Regents of University of California 16

17 Mapping Learning with Ontology- Based Design Specifications Principle DOMAINS Relevant texts Expert judgment requires Includes components Procedure related-to definition requires Concept Best Practices Fundamental principles Regents of University of California 17

18 Ontology in Use requires Dividing fractions Rate/ change in position related-to Rational Number Equivalence Velocity definition requires Variables Communication Design of Assessments Scoring criteria & performance standards Analysis and feedback to designers, policy makers, educators IMPLEMENTATIONS Regents of University of California 18

19 Regents of University of California 19

20 POWERSOURCE Foundations Expert knowledge is schema-based, organized around complex, interconnected concepts Expert knowledge organization supports problem solving, inferences, transfer to other content Experts provide the basis of material design, scoring, and formative assessment (feedback) Application of what we know about experts to support content area expertise Regents of University of California 20 20

21 Common Core Ontology in Math Regents of University of California 21

22 Cognitive Models for Assessment and Interventions Regents of University of California 22 Regents of the University of California

23 Expanded 21st Century Skills Cognitive, Social, Intrapersonal Learning static.flickr.com/80/ _055b631d8c_b.jpg Adaptive, complex problem solving Situation awareness and risk assessment Decision making Self-regulation Teamwork Learning to learn Communication Social-emotional learning Conceptual, procedural, and systemic learning of content Regents of University of California 23

24 Problem Solving Ontology Identify problem(s) Evaluate solution(s) Access knowledge, Monitor self Use feedback Represent problem(s) Plan solution(s) Regents of University of California 24

25 Problem Solving Ontology Regents of University of California 25

26 Design Features and Validity Examples 1. Learning Games 2. Teacher Evaluation 3. Large Scale Assessment Regents of University of California 26

27 Technology-Based Games Regents of University of California 27

28 Performance Expectation 5-8 Year Olds Plan and conduct an investigation and give evidence of the effects of balanced and unbalanced forces on the motion of an object Unbalanced forces: Changes in velocity Regents of University of California 28

29 Game to Teach Social-Emotional Learning To recognize and accurately label own and others emotions Features: facial variations, team discord, teamwork to solve problem, space narrative, scenario Regents of University of California 29

30 We Learned: General Feature Analysis Design features comprise any game, instruction, or assessment: Content, e.g., topics or operations Language, e.g., complexity, syntax, vocabulary Representation(s) graphics, pictures, text Narrative or scenario Directions and purpose Task formats and response modes If structured systematically, games can be designed faster and with quality Regents of University of California 30

31 Features in Game Design have Additional Uses to Understand Effects Game levels tagged with features to monitor students progress in learning Game features represented in assessments Assessment items tagged for analysis using Big Data approaches Features identified that have the most impact on learning Regents of University of California 31

32 Assessment of Outcomes Most think of format, multiple-choice, essay Content Cognition For individuals, groups, programs, institutions Core of data for educational evaluation Regents of University of California 32

33 Assessment Depends on Appropriate to purpose(s) Transparency to teachers Adequate content representation Specific cognitive demands Logical and clear scoring Reporting in a form helpful to users for making decisions Regents of University of California 33

34 Types of Assessments Product, performance, combination Ephemeral, recorded, reported Independent with help, modeled Collaborative or team task Extended, interactive Generated by learner with minimal helps Uniform Varied levels of complexity in tasks, scoring, interaction Cost and time drivers Regents of University of California 34

35 A Perpetual Reform Performance Assessment Highly desirable Takes a longer time to administer Normally expensive to develop and score Hard to develop good technical indicators, like comparability Regents of University of California 35

36 Development of Feature-Based Performance Assessments Domain Problem Solving Scenario Stimulus Response Regents of University of California 36

37 Rotation of Features Scenario or Narrative Concepts Cognitive Elements Task Requirements Specific Response Options Language components Regents of University of California 37

38 Feature Analysis of Existing Artifacts Designed or applied to existing instruction and assessments Tagged on tasks and analyzed with data mining Analysis helps determine the features leading to learning and those that make no contribution Regents of University of California 38

39 Validity Does intervention or assessment performs as intended? Usually applied to tests and depends upon their purposes and features Validity refers to decisions made from results Are they warranted and consistent with the purposes? Regents of University of California 39

40 Validity and Test Purposes Single purpose tests in the US - college admissions Multiple purpose tests Assessing student progress, giving accountability information, and guiding teachers to improve All three purposes would need validity evidence Regents of University of California 40

41 Example 2 - Evaluating Teachers Using Value-Added Models One logical way to evaluate teaching is to see how well their students do on examinations Controversial because all tests have errors There is a large difference of opinion between the government and academic experts on the utility of this approach Regents of University of California 41

42 Factors in Using Value-Added Models Teachers teach different children at different kinds of schools Even if performance is averaged over two or three years, there is more difference for a teacher year-to-year than between different teachers In the US, standardized tests are too general to measure what is actually taught, for there is no standardized curriculum Regents of University of California 42

43 Evaluating Teachers Using Value- Added Models Students and teachers are not randomly assigned to schools = bias Policy and leadership changes If teachers' salary or jobs depend upon student performance, consequences? Incentive for teachers to help students on the tests, or even corrupt the findings by changing answers Regents of University of California 43

44 Teacher Value-Added Model Goal to assess teacher competence using student performance Need to specify range of teachers, range of content, confidence in tests, variation in students over years Problems: Tests are too general Tests are not transparently related to goals Children in classes vary year to year Policies change Teacher and student types are confounded by classroom assignment (in US) Requires randomization of teachers and students to classes and schools. Not practical Regents of University of California 44

45 A Method to Evaluate Teachers? A method that is fair and accurate Cannot be easily corrupted Encourages teachers and students learn more Shows that we value teachers Analysis of instructional and exams using features scientifically related to learning Regents of University of California 45

46 Example 3: Feature-Based Validity Using State Level Assessment Can State leaders trust results of annual tests? Quality assessments should require high performance on content and cognition on relevant to standards Data should show whether the test or its items are sensitive to instruction, and therefore can be validly used to measure school or teaching effectiveness Regents of University of California 46

47 State Assessment Feature Processes Tagged every test item in language and math for grades 3 & 4 and 7 & 8 for years 2011, 2012, and 2013 Trained raters on features with high consistency Rated features of content, cognition, linguistics, and tasks Used to tag each item in data Regents of University of California 47

48 State Assessment Functional Validity Process Ran data to determine whether overall items changed year-to-year Used data-mining to find specified and other features associated with student growth Final feature set accounted for 50% of variation of performance Used to guide teachers and professional development to improve test performance without direct practice Regents of University of California 48

49 Summary Design goes hand in hand with validity Feature analysis integrated with data analysis and inferences Examples in learning games, teacher evaluation, statewide assessment Assessments should give information that can be used to improve learning Regents of University of California 49

50 What Does the Future Hold? More coherent learning More transparent design Supports for teaching and better materials Fair, valid and understandable tests Technology Confidence that the process is fair and trustworthy Regents of University of California 50

51 Eva L. Baker Copyright 2014 The Regents of the University of California. Do Not Distribute

52 Back Up Slides Back up slides Regents of University of California 52

53 Regents of University of California 53

54 Citations hdp://lasallecancun.edu.mx/author/admin/page/4/ hdps://nunj.mx/sep- publica- perfiles- a- evaluar- futuros- maestros/ hdp://culturaeducajva.org/?p=1826 hdp:// hdp://enviajes.cl/mexico/lugares- turisjcos- de- mexico- 2/ hdp:// 10- Lugares- turisjcos- de- Mexico.html hdp://enmexico.about.com/od/cultura- en- movimiento/tp/diez- Monumentos- EmblemAJcos- De- Ciudad- De- MExico.htm hdp:// de- desjnos/mexico- y- caribe/ciudad- de- mexico/ hdp:// city/zocalo- plaza- de- la- consjtucion- mex- df- z.htm hdp://es.wikipedia.org/wiki/bas%c3%adlica_de_santa_mar%c3%ada_de_guadalupe hdp:// stroller- scooter hdp:// hdp:// neck- blouse html p. 3: hdp://c2.ac- images.myspacecdn.com/images02/129/l_0157a4bec6d648a482fa\71d28befa9.jpg p. 6: hdp:// +pencil&gs_l=img.1.1.0l j ac. 1.8.img.i1WEwepTyC4#hl=en&site=imghp&tbm=isch&sa=1&q=black+boy+wriJng&oq=black+boy+wriJng&gs_l=img j0i j c.1.8.img.PSTWKNOryGA&bav=on.2,or.r_qf.&bvm=bv ,d.cGE&fp=83bce1ba5cf84439&biw=1003&bih=565&imgrc=ICcf_0GWMthNlM%3A%3Bu- UW7eWIxeyMbM%3BhDp%253A%252F %252Fwww.thenorthstarnews.com%252FContent%252F101%252Fblack%252520boy%252520studying.jpg%3BhDp%253A%252F %252Fwww.thenorthstarnews.com%252FIssue.aspx%253FID%253D %3B183%3B276;hDp://library.thinkquest.org/04oct/00492/ Quiz_pencil.jpg; hdp://casualgamerchick.com/2007/06/30/achievements- coming- to- casual- games/ Regents of University of California 54 Regents of the University of California Regents of the University of California