Are majors achieving our key goals?

Transcription

1 Are majors achieving our key goals? Involving faculty and students in the creation of a cumulative assessment Jenny Knight MCD Biology

2 The rest of the MCDB SEI team: Science Teaching Fellows Undergraduates/recent graduates Caleb Trujillo (now a Ph.D. student, Purdue University) Michelle Smith now at UW Sarah Wise Bre Pritchard Katie Southard MCDB SEI director Bill Wood Jia Shi

3 Background on Concept Assessments (Inventories) Most have been designed: To diagnose areas of conceptual difficulty and/or To assess changes in conceptual understanding related to a specific interventions For Intro-level students As multiple choice Astronomy: ADT Physics: FCI, BEMA Geology: GCI Biology: CINS, HPI, BCI, GCA, IMCA ( free response assessments in biology: Diagnostic Question Clusters (conservation of energy and conservation of matter):

4 Concept assessments are also often used as a pre-post assessment: Independent (from other course assessments) measure of conceptual understanding Compare the correct/incorrect ideas students have on 1 st day vs. last day of course (same questions) Calculate gain

5 This project is unique in that it will attempt to design a tool for measuring student understanding at the culmination of their studies This project also requires the expertise of faculty, the opinions and knowledge of students, and collaboration among everyone to determine goals that encapsulate a whole major

6 Development Process Decipher what students think about high level concepts in biology, and use these ideas as distracters in the assessment Step 1: Establish Themes and Learning Goals Identify concepts that run through the major: introduced in early courses, and built on throughout the consecutive courses Discuss with faculty their overall goals for students Establish a set of learning goals that faculty agree reflect what a graduating senior should be able to do in the area of molecular, cell and developmental biology. (based on Treagust, 1988 and others)

7 Step 2: Interview Students Using the goals as a guide, ask students open-ended questions about core concepts, especially ones known to be challenging Use student responses to identify which concepts really are challenging, and how they are challenging Step 3: Create multiple choice questions Focus primarily on concepts that students have difficulty with Create questions with distracters and correct answers from student supplied responses The idea is that if the distracters truly represent student thinking, we will get a clear picture of the most common incorrect ideas from student responses

8 The process is iterative and involves conversations with faculty and students at each point Step 4: Administer pilot assessment and interview additional students. Revise assessment. Step 5: Solicit evaluations of the questions from internal and external faculty. Revise assessment. Step 6: Repeat the student interview and faculty evaluation process. Step 7: Re-administer assessment and perform statistical analyses to determine evidence of validity and reliability of assessment

9 Faculty involvement: defining the content Add learning goals that are missing Review learning goals and help create sublearning goals Contribute and review questions Re-review after revisions

10 Student involvement: refining the content Ask students questions about fundamental concepts Use their ideas to generate distracters Interview students using pilot multiple choice questions to see if their answer choices reflect their reasoning Repeat if necessary

11

12 We have a draft set of broad Overall Goals for graduating MCDB majors* 1. Integrate and apply concepts from previous physics, chemistry, and MCDB core courses to a new, complex system. 2. Distinguish between and justify the use of different possible experimental approaches and predict possible outcomes of such experiments. 3. Interpret and draw conclusions from graphical and pictorial data. 4. Evaluate conclusions from historic and current literature. 5. Convey the fundamental concepts of molecular, cell and developmental biology to others. *Arrived at through interviews with 9 MCDB faculty, many informal conversations, and the learning goals of all of the MCDB core courses

13 We have some Content Learning Goals and multiple choice questions (work in progress) Three broad areas 1. Genetics and gene expression (4 sub goals; 13 questions) Example: Predict how different kinds of mutations in a gene can lead to particular phenotypes based on the known function of the gene. 2. Roles of signaling molecules (4 sub goals; 5 questions) Example: Predict ways in which the binding of an extracellular ligand to a cell surface receptor could affect the expression of a particular gene in the nucleus of that cell, and how the outcome could be different in different cell types 3. Evolution (3 sub goals; 2 questions) Example: Defend the view that evolution proceeds as the result of random mutations.

14 Student involvement: refining the content Ask students questions about fundamental concepts (8 students interviewed) Use their ideas to generate distracters Interview students using pilot multiple choice questions to see if their answer choices reflect their reasoning (12 students interviewed) Administer 20 pilot questions to 111 students in capstone course

15 Pilot version of 20 questions to students entering capstone courses in Fall 2010 (Developmental Biology, n=111) For 13 of the 20 questions, the average was less than 50% correct 5 were re-asked on the final, and student comments solicited On 2 of these, the students now scored over 70% correct On 2, students continued to struggle, but also revealed some elements of the questions that could be improved to eliminate confusion On 1, students still clearly struggle with the concept Conclusions: some of our questions may be at the right level and may be identifying persistently difficult concepts some of our questions need work (too easy or not wellworded) we need more questions on certain topics

16 Faculty involvement (9 initially interviewed on learning goals)

17 Now: get faculty to contribute ideas and questions Norm Pace Evolution What should students be able to do on this topic? Where do they typically get things wrong? What are some examples of questions you like to ask that students struggle with? Planned: meet as a group to discuss learning goals What is missing? Kevin Jones Plasticity Joaquin Espinosa Signaling

18 Conclusions In generating and using a capstone level Concept Assessment, we will be able to: 1. Identify persistent incorrect ideas, still present after 3-4 years of college biology instruction (entering final capstone courses) 2. Measure learning gains at end of capstone courses 3. Inform faculty of results, which can -help inform possible changes in course pedagogy -facilitate broader discussion of curricular change

19

20

21 Usual sequence of core courses for MCDB majors Year Fall Spring Freshman Introduction to Molecular and Cell Biology (assessment: IMCA) Principles of Genetics (assessment: GCA) Sophomore Cell and Molecular Biology I Cell and Molecular Biology II Junior or Senior year (choose one capstone course) Developmental Biology or Immunology or Molecular Neurobiology The CONTENT of the final capstone courses is not the same but there are overlapping CONCEPTS

22 General Topic Genetics and Gene Expression Questions on pilot 1,2,3,4,5,6,7, 13,14,15,16, 17,24 (13 questions) Questions for which % correct was <50 (pre) 1,2,3,6, 14,15,16, 24 Potential misconceptions and failures in conceptual connections (from interview results and selection of answers on pilot) Stop codons stop mrna production as well as protein production Alternative splicing happens during mrna production and primarily affects inclusion of introns A change to the regulatory region of a gene affects the function of that gene Fail to consider possible effects of epigenetics, such as X inactivation, on phenotypes Signaling and Cytoplasmic Determinants 8,9,11,12,16 (5 questions) 8,9,11,16 Only signaling (not differences in cytoplasmic contents of a cell) affects a cell s identity Inhibition of activity of a protein is the same as inhibition of production of a protein Failure to connect effects of a signaling on transcription and eventual phenotype of cell Evolution 19,23 (2 questions) 19, 23 Mutations that confer resistance or improve fitness occur in response to environmental change rather than randomly

23 Should such an assessment be multiple choice, free response, or both? Free response allows for collecting more information about what students think Text analysis software (collaboration with Mark Urban- Lorain, John Merrill, Julie Libarkin (MSU) and Ross Nehm (OSU), NSF DUE ) But, multiple choice allows for faster and easier analysis, and is more transportable to other institutions Theoretically, a multiple choice instrument built using student-supplied ideas will help capture what students are thinking