Program Learning Outcomes Data Analysis/Reporting Workshop. Center for Teaching & Learning AUB February 10, 2011 Facilitator K. El Hassan, PhD.

Transcription

1 Program Learning Outcomes Data Analysis/Reporting Workshop Center for Teaching & Learning AUB February 10, 2011 Facilitator K. El Hassan, PhD.

2 Workshop Outline Introduction Recap Scoring and Analysis Tips Reporting Guidelines & Templates Exemplars

3

4 Examples of Assessment Approaches Available Data Assessment tool? Who or what is assessed? What can be assessed Self-reports Classroom assessment Focus groups Interviews Phone surveys or interviews Reflective essays Surveys (local or standardized) Alumni Employers Enrolled students Faculty Graduating students Off-campus supervisors Parents Staff Perceptions about: Campus climate Evaluation processes Perceived learning Educational outcomes Attitudes Values Achievement tests Test score analysis Content analysis Scoring rubrics Competitions Embedded questions on exams Locally-developed exams Oral thesis defense Orals exams, recitals Standardized tests Mastery and knowledge of principles, skills Value-added Observations Case Studies Observations Campus events (sports, theater) Classes Club meetings Faculty offices Fieldwork sites Student services offices Attitudes Campus climate Interactions Processes Services Student involvement Student learning Student academic work Content analysis Scoring rubrics Capstone course products Homework papers Portfolios Presentations Performances Publications Research reports Term papers, Theses Videotapes Mastery and knowledge of principles, skills Values Processes Value-added Campus documents Course x program objectives matrix Administrative units Departments Accuracy Cohesion/consistency

5 II. Do. Step 7. Collect Data This phase involves measuring outcomes. When will measurements be made? How often will measurements be made? Who (specifically) will collect the data? Who will interpret the data? Where/When will data be interpreted? Who will report the results? To whom will results be reported? By when will results be reported? 5

6 PLO matrix (cont d) Learning Outcome Course Year 1 Year 2 Year 3 Fall Spring Fall Spring Fall Spring 4- Apply evidence-based medical nutrition therapy and critical thinking in the Nutrition Care Process. NFSC 240 NFSC 295 NFSC 294 D NFSC 240 D NFSC 295 D NFSC 294 I Focus Group Interview 5- Integrate nutrition principles and communication skills in nutrition education for groups and communities. NFSC 222 D NFSC 222 I Focus Group Interview 6- Be able to communicate effectively and maintain lifelong learning behavior, integrity and empathy in professional practice. NFSC 296 MNGT 299 D NFSC 296 NFSC 299 I Employer & Graduate School Surveys D: Direct assessment, I: Indirect assessment Course embedded questions: 80% of students answer correctly 80% of the selected pool of embedded questions (final exams): 20% of final exams in selected courses should consist of embedded questions, previously agreed upon by the major in consultation with the instructor Capstone course (NFSC 299): 80% of the students score above 80 in their final course grade

7 Step 8: Analyze the results Faculty should be the ones responsible for the analysis and interpretation of data. It is important to summarize the results in a meaningful way so that they can be reviewed and actions needed to improve the program can be determined. Need to keep in mind the audience when analyzing results like who will access and use the data, and accordingly need to vary analysis and reporting procedures according to the identified audience 7

8 Scoring Tips Need to prepare clear answer key of acceptable answers for each question. Objective Questions Group groups of questions that reflect a specific dimension of outcome By grouping within a dimension, faculty will enter one number for dimension, i.e. number correct out of say 5 questions. Short Answer Every instructor must score their students based on common answer key, though they are free to grade additional answers as correct in determining their own course grades.

9 Scoring Non-Objective Assessments Identify the dimensions of the outcome i.e., the key broad aspects or areas of student performance on which the assessment will be scored. Ex: Analyzing and interpreting data: identify and use data analyze and interpret data draw conclusions based on analysis and interpretation of data Develop a rubric that specifies level of achievement (exemplary, satisfactory, unsatisfactory) for every dimension of the outcome Course instructor grades paper (project, etc) using rubric. Random sample of papers collected by assessment committee

10 Assessment Reporting The first step in developing an assessment report is to answer the following questions: 1. Who is the audience for this report? 2. What do they want to know? At its most basic, report should have information to answer five basic questions: 1. What did you do? 2. Why did you do it? 3. What did you find? 4. How will you use it? 5. What is your evaluation of the assessment plan itself? 10

11 Sample Report Description On an annual basis, programs are required to identify and assess at least one intended outcome. Although assessment of only one outcome per year will be required, all outcomes must be assessed prior to the program s scheduled program review. The annual assessment report to contain the following components 1. Intended outcome statement identified for assessment 2. Summary of assessment methods/measures to include brief explanation of methodology, persons responsible for data collection and analysis, and a summary of assessment results. 3. A description of the process used to disseminate assessment results to key stakeholders and summary of changes or improvements proposed based on assessment results. 4. A brief description of the plan and timeline for implementing the proposed change (s) and any resources needed to implement the proposed change (s), using existing resources already in place, reallocating existing resources from other areas, or allocation of new funds. Western Carolina University Program Assessment Handbook 11

12 Adapted from Departmental Guide and Record Book for Student Outcomes Assessment and Institutional Effectiveness, ASSESSMENT REPORT FOR (Instructional Degree Program) (Degree Level) (Assessment Period Covered) (Date Submitted) Intended Educational (Student) Outcome: NOTE: There should be one form C for each intended outcome listed on form B. Intended outcome should be restated in the box immediately below and the intended outcome number entered in the blank spaces. First Means of Assessment for Outcome Identified Above: a. Means of Program Assessment & Criteria for Success: a. Summary of Assessment Data Collected: a. Use of Results to Improve Instructional Program: Second Means of Assessment for Outcome Identified Above: b. Means of Program Assessment & Criteria for Success: b. Summary of Assessment Data Collected: b. Use of Results to Improve Instructional Program: 12

13 Due Annually by Western Carolina University (Program Name) (College Name) Annual Assessment Report for 200_-200_ Primary Contact Name/Info: Provide name and contact info for person responsible for submitting program assessment plan. Intended Learning Outcome to be Assessed this Cycle State the outcome(s) that the program has chosen to assess this annual cycle. Method(s) of Assessment Provide a summary of the methods of assessment used to assess the chosen outcome. Note any changes in the assessment measures from the program s official assessment plan. Results of Assessment Results must include a summary of major findings, interpretation of the results, and a description of how the results were disseminated to key stakeholders for discussion. Implementation Plan Identify what programmatic or curricular changes, if any, you will make as a result of the assessment results. Each recommended action must be specific and relate directly to the outcome and results of assessment. A description of the timeline for action and the person(s) responsible must be included. In addition, please include a brief description of resources that will be critical to implementation of the actions proposed, if applicable. 13

14 Exemplar: Information Literacy SLO: Assessment Instrument: Scoring: Students will be able to integrate information from different types of secondary sources to support a thesis on a research topic. Students will write an 8-10 essay using sources to support their thesis/claim The student s response will be assessed using a rubric measuring effective use of Works Cite or Bibliography, Identifying Sources, and Use and Documentation of sources. Each dimension is assessed on a scale of excellent (3), satisfactory (2) or unsatisfactory (1) Data Collection Column Heading Effective Use of Works Cited or Bibliography Identifying Sources Use and documentation of sources Min/Max Score 1=Min, 3=Max 1=Min, 3=Max 1=Min, 3=Max Satisfactory Score Score 2 or more 2 2 or more 1 2 or more 2

15 Scoring Rubric Criteria Excellent Satisfactory Unsatisfactory Score Effective Use of Works Cited or Bibliography 2 Identifying Sources Use and documentation of sources Has complete, accurately formatted works cited or bibliography page Correctly selects from popular and academic sources, considering currency and accuracy. Summarizes or paraphrases source material in own words or quotes verbatim sparingly, using proper in-text citations. Works cited or bibliography page is complete but has some format or content errors Usually, but not always, correctly chooses current or accurate popular and academic sources. Sometimes summarizes or paraphrases source material in own words but quotes verbatim material too often. Uses intext citations, but sometimes has inconsistent formatting. Works cited or bibliography page is incomplete: May be missing entries or information or has many format errors. Chooses inappropriate or outdated sources for the given assignment Does not summarize or paraphrase source material accurately; uses verbatim quotes extensively with inaccurate or no intext citations. 1 2

16 Exemplar: Effective Communication SLO: Assessment Instrument: Scoring: The student will be able to write multiple-page essays that meet college-level academic standards for content, organization, style, grammar, mechanics, and format. Students will write an 8-10 essay using sources to support their thesis/claim The student s response will be assessed using a rubric measuring mechanics and style, organization and development. Each dimension is assessed on a scale of excellent (3), satisfactory (2) or unsatisfactory (1). Data Collection Column Heading Min/Max Score Satisfactory Score Score Mechanics & Style 1=Min, 3=Max 2 or more 2 Organization 1=Min, 3=Max 2 or more 1 Development 1=Min, 3=Max 2 or more 1

17 Criteria Excellent Satisfactory Unsatisfactory Score Mechanics & Correct topic choice, Style length and format. Adequate topic choice. Close to correct length. May include some formatting errors (e.g. nonstandard font size or margins). Inappropriate topic choice. Far exceeds or falls short of required length. Does not follow required format. Few or no sentence structure, grammar, spelling, or punctuation errors. Few sentence structure, but some grammar, spelling, or punctuation errors, but these don t interfere with the reader s comprehension of the paper. Sentence structure, grammar, spelling, or punctuation errors significantly hamper comprehension. Organization Appropriate use of diction and word choice Thesis (claim) is clearly stated based on focused research question(s). Paper is logically organized. Main ideas are clear and supported in well constructed paragraphs with transitions. Some diction or word choice is inappropriate or informal Thesis is less clearly stated. Paper has basically logical organization. Most paragraphs are well constructed; main ideas are evident in most. May have limited use of transitions Diction or word choice is consistently inappropriate or informal. Thesis is not clearly stated. Little or no overall organization. Paragraphs may be poorly constructed. Or confusing with few or no transitions. Development Intro and conclusion are comprehensive and thesis driven Paragraphs use relevant evidence, such as examples, paraphrases, or quotes to support thesis. Substantive and factual information is used throughout. Intro and conclusion satisfactorily address the thesis. Each paragraph has some relevant examples, paraphrases or quotes as evidence to support the thesis. Essay has relevant content, but may include fluff or nonrelated ideas. Intro and/or conclusion are missing or don t address the thesis. Paragraphs have few, if any examples, paraphrases or quotes as evidence. Much of essays content is irrelevant or immaterial

18 General Education Outcome. Students will discover how they express values and beliefs through actions and learn to evaluate the effects of values and beliefs upon their daily lives Method of Assessment: The outcome was addressed via two essays: one on the first course exam and one on the final. The first question asked students to discuss how the task roles they enact reflect their values and beliefs (4 pts.) and explain the effects of their task roles on the group s interaction (6 pts.). The final question asked students to identify their ethical orientation (discussed in class) (2 pts.), provide examples of how they communicated this ethical orientation in their semester-long small group (6 pts.) and describe how their communication of ethical orientation influenced the group as a whole. (6 pts). Point values for each section of the question appear in parentheses. Student mastery of the general education outcome was based on the entire score (max. of 10 pts. on first question and 14 pts. on final question), as the outcome requires students to demonstrate both how values and beliefs are expressed through their actions and to evaluate the effects of those values and beliefs on their daily lives.

19 Summary of Assessment Evidence Collected Number of student essays (first exam) Category 4 (16.6%) Exceeds Expectations (90% or above) 13 (54.2%) Meets Expectations (75%-89.9%) 7 (29.2%) Does Not Meet Expectations (below 75%) Number of student essays (final exam) Category 4 (16.7%) Exceeds Expectations (90% or above) 15 (62.5%) Meets Expectations (75%-89.9%) 5 (20.8%) Does Not Meet Expectations (below 75%) Conclusions Drawn from Assessment Evidence Collected: More than 70% of students (70.8% first exam and 80% final exam) achieved the outcome by meeting or exceeding expectations, which the department found acceptable. For students who did not meet expectations, the difficulty appeared to be identifying how task roles reflected values/beliefs and specific examples of how they communicated ethical orientation

20 General Education Category Science Learning Outcome: Students will develop an understanding of the methods the natural sciences use to study the physical world, through courses that include an experimental component and/or an observation component and/or a component to evaluate data and develop and test hypotheses. Method of Assessment: Outcome (c) was assessed via a diagnostic exam developed by the Physics Education Research community: the Views on the Nature of Science (VNOS). Students respond to 25 questions about science and their responses are rated as: naïve thinking about science, competent thinking about science, or sophisticated thinking about science. Each student can then be categorized as naïve thinker, competent thinker, or sophisticated thinker. Outcome (a) is assessed using a different measure and is reported on a separate sheet.

21 Summary of Assessment Evidence Collected Number of students Category 6 (9%) Sophisticated thinkers about science 26 (40%) Competent thinkers about science 33 (49%) Naive thinkers about science Successful completion of outcome (c) is considered students who are competent or sophisticated thinkers about science. Conclusions Drawn from Assessment Evidence Collected : 49% of the class achieved competent or sophisticated thinking about science. This is above national averages for general education science courses as reported in Physics Education Research literature. How the Conclusions Influence the Assessment Process and/or Future Offerings of This Course

22 Exemplar: Nutrition & Dietetics Learning Outcomes Demonstrate an understanding of the metabolism of macro and micro-nutrients under different physiological and pathological conditions Be able to communicate effectively and maintain lifelong learning behavior, integrity and empathy in professional practice. Direct assessment methods in form of course embedded questions in final exam (20% of final) in NFSC 240& 274. Criteria: 80% of students answer correctly 80% of the selected pool of embedded questions

23 Suggested Scoring, Analysis & Reporting Questions 1-5 are marked items for assessing outcome. Student average on combined five items calculated, and ranked in ascending order. Frequency of scores tabulated. Percentage of students with 80 and above is calculated. Check if results meet preset criteria of 80%

24 Formal Reports/ Oral Presentations Identify dimensions required by learning outcome. Develop scoring rubric according to dimensions assigning weights to each component/dimension. Instructor scores report, essay, report according to rubric. Report presented based on attaining preset criteria or not.

25 Communication & Media Effectively use old and new media skills, technologies and concepts to plan, research, create, communicate and disseminate targeted messages and communication material through various media, and to execute full media campaigns for advertising, public relations and political communication purposes. Methods: two courses Question 1 on final exam Project Criteria 70% on question on and on project Results. Did not meet it on exam but on project yes.

26 Suggested Scoring/Analysis For final exam question, take score on question and check if average is 70, or that all students obtained 70 or more (preferable). For project, either holistic scoring can be used to evaluate project and check if average on it is 70 or all students obtained 70 and above, or analytic scoring can be used (preferable more informative) with various dimensions identified and scored.

27 Thank You