13 th Annual Texas A & M University Assessment Conference College Station, Texas February 2013

Transcription

1 13 th Annual Texas A & M University Assessment Conference College Station, Texas February 2013 Melissa B. Weston, Chair, Department of Psychology El Centro College, Dallas, Texas mweston@dcccd.edu David Goomas, Psychology Faculty El Centro College, Dallas, Texas dgoomas@dcccd.edu

2 BEFORE THE RUBRIC(S) CAME THE STANDARDS I. The Principles of Accreditation: Foundations for Quality Enhancement of the Southern Association of Colleges and Schools Commission on Colleges (SACSCOC) (SACSCOC, 2012) The SACSCOC establishes standards for the accreditation of colleges and schools within its purview. Basic requirements are those detailed in SECTION 2: CORE REQUIREMENTS SECTION 2.5: The institution engages in ongoing, integrated, and institution wide research based planning and evaluation processes that 1. Incorporate a systematic review of institutional mission, goals, and outcomes, 2. Result in continuing improvement in institutional quality, 3. Demonstrate the institution is effectively accomplishing its mission (Institutional Effectiveness). 2

3 THE PRIMARY OBJECTIVE ACCORDING TO SACSCOC IS SECTION 3: COMPREHENSIVE STANDARDS Section 3.3.1: Institutional Effectiveness: The institution identifies expected Student Learning Outcomes (SLOs)*, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional Effectiveness) (Southern Association of Colleges and Schools Commission on Colleges, 2012) Educational programs, to include Student Learning Outcomes THE PRIMARY OBJECTIVE ACCORDING TO THE THECB** IS The Undergraduate Education Advisory Committee (UEAC) of the THECB approved the following OBJECTIVES AND COMPONENT AREA MAPPING on April 16, Edits were approved by the THECB on October 27, 2011 (See Handout 2, UEAC Objectives and Component Area Mapping). (Passed into Law by the Texas Legislature, November 2011) * See Handout 1: Psychology Curriculum Map ** Texas Higher Education Coordinating Board 3

4 Objectives and Component Area Mapping* Approved by UEAC (Edits approved ) Foundational Component Areas Critical Thinking Communication Skills Empirical & Quant Skills Teamwork Social Responsibility Personal Responsibility Communication X X OPTIONAL X OPTIONAL X Mathematics X X X OPTIONAL OPTIONAL OPTIONAL Life & Physical Sciences X X X X OPTIONAL Language, Philosophy, and Culture X X OPTIONAL OPTIONAL X Creative Arts X X OPTIONAL OPTIONAL X American History X X OPTIONAL OPTIONAL X Government/Political Science X X OPTIONAL X X Social/Behavioral Science X X X OPTIONAL X Institutional Option* OPTIONAL OPTIONAL OPTIONAL OPTIONAL OPTIONAL X = Required Core Objectives to be addressed in each course selected in the Component Area. Optional = Institution may include Core Objective for each course selected in the Component Area. (See HANDOUT 1, CURRICULUM MAP PSYCHOLOGY.) 4

5 THE EL CENTRO COLLEGE (ECC) QUALITY ENHANCEMENT PLAN (QEP) TIMELINE 2010 El Centro College QEP topic identification process begins with submission of White Papers on topics of consideration. June 2011 QEP Topic Committee established. Fall 2011 QEP Topic Critical Thinking was identified as El Centro College s QEP topic BEFORE its identification by UEAC as one of six Component Areas* to be taught across the curriculum (See Handout 2). El Centro College defines critical thinking as, a disciplined and continuous process of asking the right questions and practicing logical thought processes to come to justifiable conclusions. ** *Handout 2: UEAC Objectives and Component Area Mapping **QEP Topic Committee, October

6 ASSESSMENT: WE KNOW HOW BUT IT ISN T ALWAYS EASY. PROBLEMS ENCOUNTERED IN IMPLEMENTATION OF NEW PROCEDURES REQUIRED BY SACSCOC AND THE THECB Faculty buy in Resistance to change CAVE Dwellers (Colleagues Against Virtually Everything) We ve ALWAYS done it this way WHY CHANGE NOW? WHERE TO BEGIN Curriculum Maps Course and Program Levels aligned with maps (HANDOUT 1) Identify Student Learning Outcomes (SLOs) (See Handouts 1 and 2) Consult with Departmental Faculty insure shared goals Develop Rubric(s) (HANDOUT 3*) together thus assuring OBJECTIVITY, NOT SUBJECTIVITY Meet regularly to compare individual faculty/course outcomes *Handout 3: Analytic Critical Thinking Rubric, El Centro College Department of Psychology 6

7 WHAT DO WE OR DO WE NOT ASSESS? DO ASSESS Student Artifacts (i.e., Exemplars) What constitutes an Exemplar? That depends on your discipline. Written assignments Essays homework or questions embedded in exams Critiqués and Term Papers Psychology, other Social Sciences, and Humanities Assessments of topical issues that require critical analysis for completion Performance Art Musical performances and compositions Art work original paintings, portfolios, etc. Original Interior and Architectural designs Theatrical performances 7

8 WHAT DO WE OR DO WE NOT ASSESS? Continued Vocational Work Original Pattern construction Garment construction Exemplars of assignments are the history of INDIVIDUAL STUDENT PERFORMANCE AND PROGRESSION OF LEARNING. Quality of Exemplars can be measured that is what RUBRICS are for. You DO NOT ASSESS Course Grades because Grades are a number and reveal no specific data. Grades have no history and do not show WHAT is being or has been assessed. How about an EXAMPLE of a rubric CURRENTLY in use? "Ask and it will be given to you. (The Gospel according to St. Matthew) 8

9 El Centro College: Psychology: Assessment of Critical Thinking Rubric Rater (Scorer): Rebecca Dryden-Shepherd Class/Course/Section: PSYC 2301/53xxx Student ID: Date: 25 POINTS TOTAL *A rating of 3.5 indicates an average (70%) understanding of evaluation material Whether knowingly or unknowingly plagiarism will result in an automatic grade of zero (Ø). Performance Element I. Knowledge And Comprehension Exemplary (5) Consistently demonstrates clear, accurate, detailed and comprehensive understanding of the relevant facts/data/theories/ concepts as well as the ability to organize the information for application, presentation and documentation Elaborates fully on topic & uses clear, effective examples. Proficient (4) Demonstrates an adequate understanding of the relevant facts/data/theories/ concepts as well as the ability to organize the information for application, presentation, and documentation Elaborates on topic but does not clarify using supporting examples. Developing (3) Demonstrates an uneven and shaky understanding of the relevant facts/data/ theories/ concepts as well as a limited ability to organize the information for application, presentation, and documentation Provides minimal support, with little explanation. Emerging (2) Demonstrates an inadequate understanding of the relevant facts/data/ theories/concepts as well as a limited ability to organize the information for application, presentation, and documentation Elaboration of evidence is insufficient or inaccurate. Minimally or Not Present (1 or 0) Does not identify relevant facts/ data/ theories/ concepts and does not demonstrate the ability to organize information for application, presentation and documentation Lacks evidence to support claims, essentially an opinion assignment OR answer is completely inaccurate. Comments Score Comments: N/A II. Application And Analysis Demonstrates confident ability to work with the key concepts/ research/ theory applying or extending them to a wide variety of new problems or contexts, making predictions, raising questions, recognizing hidden assumptions, drawing inferences, analyzing solutions, communicating insightful contrasts and comparisons. Demonstrates adequate ability to work with the key concepts/ research/ theory applying or extending them to a wide variety of new problems or contexts, making predictions, raising questions, recognizing hidden assumptions, drawing inferences, analyzing solutions, communicating insightful contrasts and comparisons. Demonstrates uneven and shaky ability to work with the key concepts/ research/ theory applying or extending them with mixed success to new problems or contexts, making predictions, raising questions, recognizing hidden assumptions, drawing inferences, analyzing solutions, communicating insightful contrasts and comparisons. Demonstrates limited ability ability to work with the key concepts/ research/ theory applying or extending them with limited success to new problems or contexts, making predictions, raising questions, recognizing hidden assumptions, drawing inferences, analyzing solutions, communicating insightful contrasts and comparisons. Demonstrates extremely limited ability or the lack of ability to work with the key concepts/ research/ theory applying or extending them with very limited success or the lack of success to new problems or contexts, making predictions, raising questions, recognizing hidden assumptions, drawing inferences, analyzing solutions, communicating insightful contrasts and comparisons. Comments: N/A Rubric developed by Associate Professor Rebecca Dryden Shepherd, El Centro College, January

10 Obtaining Faculty Buy in May be a Difficult Issue. Faculty know HOW to assess. What they do not know or are not doing is TRACKING THAT ASSESSMENT PROCESS. Faculty like everyone may be resistant to change. I don t have time to do this. Yes, you do. STREAMLINE THE PROCESS. DEVELOP OR BEG/BORROW/STEAL A RUBRIC. The Cavalry IS on the way. American Psychological Association (APA) Sources for Rubrics cyberguide v2.pdf St. Petersburg College (SPC) Sources for Rubrics Rcampus Rubrics for ALL disciplines may be imported and used as is or modified to suit your institution and department. and Rubistar Searchable rubrics with instructions on making them interactive. CAVE Dwellers Whatever you do, they will not like it. Accept that as a reality, and move on. We ve ALWAYS done it this way! Why should we change NOW? HINT: BECAUSE THIS WAY IS NOT WORKING IN THE 21 ST CENTURY! 10

11 PROCESS : Revised Assessment in the 21 st Century Curriculum Maps* provide an OVERVIEW of what each discipline wants its students to know and do on completion of a course or a program. ALL courses in a discipline are listed in sequence. Program SLOs are listed at the top of the Map, individual courses in the program are listed down the left side. Each course indicates whether a given SLO is Introduced (I), Reinforced (R), or taught at an Advanced (A) level. Each SLO is indicated as being Explicitly (X) or Implicitly (I) stated in the Course Syllabus as a Learning Outcome. Instructor Feedback (F) via Rubrics Is provided to the student on assessments (homework, projects, term papers, etc.) for each SLO. (See Handout 3) New Student Learning Outcomes for the State of Texas are provided by UEAC and will take effect with the Revised Core Curriculum beginning in the 2014/2015 academic year. Rubrics incorporate SLOs from program and course levels (SEE HANDOUT 3) Consult with Departmental Faculty in refining SLOs for your Department; these will be incorporated into your Rubric. * See Handout 1: Psychology Curriculum Map 11

12 PROCESS : Revised Assessment in the 21 st Century, continued. Establish regular, anticipated faculty meetings, to include Adjuncts Determine HOW faculty are currently assessing discipline specific SLOs Begin search for rubrics (see Slide 10) Discuss and agree on SPECIFICS for assessment of each SLO What is important? Psychology is my discipline, thus the rubric used here reflects ECC s SLOs for Psychology. Critical Thinking (this is also El Centro College s Quality Enhancement Plan (QEP) Empirical and Quantitative Skills (Scientific Methodology) Communications Skills (written and oral) Social Responsibility (we are a green institution, thus decreasing our carbon footprint is important to us) Information Literacy (online research in professional databases, ability to use computer software such as MS Office and ability to use APA Style in written work) Faculty agreement on scoring IMMEDIATE FEEDBACK TO STUDENTS our Interactive Rubric in the ecampus platform (BlackBoard 9.1 ) (See Goomas, 2012) 12

13 The Five Questions of Assessment (Jackson, R. and Johnson, M., 2007) 1. WHAT are we assessing? Student Learning Outcomes (SLOs) 2. What TOOLS will we use to gather data? Rubrics and embedded exam questions 3. From WHOM and HOW will we gather data? Data derive from students via options A, B, and C below and are determined using our Assessment Tool our Analytic Rubric A. Written Research Assignments using APA style B. Embedded Exam Questions (multiple choice, true/false, matching, essay) C. Departmental Exit Exams 4. HOW will we analyse results? A. Through use of our Critical Thinking Rubric for homework assignments and term papers (see Slide 9 and Handout 3) B. Through end of term and end of year analysis of individual class and department wide outcomes. Success is course completion with a final grade of A, B, or C. 13

14 The Five Questions of Assessment, continued (Jackson, R. and Johnson, M., 2007) 5. HOW will we USE RESULTS of assessment to IMPROVE STUDENT LEARNING? A. Twice yearly faculty meetings to analyse and discuss results B. Evaluation by departmental faculty of what worked or did not work C. Revise classroom methods and assessments as required D. Repeat the entire process EACH ACADEMIC YEAR 14

15 References Dryden Shepherd, Rebecca. El Centro Psychology: Assessment of Critical Thinking Rubric. January Dallas, TX. Jackson, R., & Johnson, M. (Presenters). (2007, April 27). Assessing student learning. Lecture presented at the SACSCOC Small College Initiative, Atlanta, GA. Johnson, M. (Presenter). (2011, April 21). Accreditation and institutional effectiveness Hidden resources. Lecture presented at NCICU Assessment Conference, Methodist University, Fayetteville, NC. The Gospel According to St. Matthew (Ed.). (1984). The New Testament: The Holy Bible, Matthew 7:7 (New International Version). Ask and it will be given to you; seek and you will find; knock and the door will be opened to you. The principles of accreditation: Foundations for quality enhancement, 5th Ed. (2012). College Delegate Assembly, pp Southern Association of Colleges and Schools Commission on Colleges, Decatur, GA 15

16 References, continued. Undergraduate Education Advisory Committee (UEAC) [Fact sheet]. (2011, April 16). Retrieved October 21, 2012, from Texas Higher Education Coordinating Board Undergraduate Education Advisory Committee website: BAA3 D2A2 3ADD779601D9B263 Weston, M. B. (2012, February 29). El Centro College Curriculum Map Department of Psychology [Curriculum Map; Microsoft Excel]. Weston, M. B., & Dryden Shepherd, R. (2012, January 30). Psychology Assessment of Critical Thinking Rubric [Chart; Microsoft Word]. Weston, M. B., & Goomas, D. T. (2012). Establishing performance standards in psychology using standardized rubrics: Meeting Texas Higher Education Coordinating Board objectives at El Centro College. Unpublished raw data, El Centro College, Dallas, TX. 16

17 Integrating Real time Antecedent Rubrics via Blackboard 9.1 into a Community College General Psychology Class David Goomas El Centro College

18 Stevens and Levi (2005) provided 6 reasons to use rubrics: 1) to provide timely feedback 2) to prepare students to use detailed feedback 3) to encourage critical thinking 4) to facilitate communications with others 5) to help instructors refine teaching methods 6) to provide a standard to fairly evaluating all students on the same assignment. Stevens, D. D., & Levi, A. J. (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback and promote student learning. Sterling, VA: Stylus Publishing.

19 * Department studies website * * Spreadsheet imbedded into the assignment * My favorite paper * Interactive rubric Blackboard 9.1 implemented the antecedent interactive rubric

20 Background: From organizational behavior studies providing the workforce with feedback to complete a unit of work improved labor standards.

21 Goomas, D. T., & Ludwig, T. D. (2007). Enhancing incentive programs with proximal goals and immediate feedback. Journal of Organizational Behavior Management, 27(1), doi /J075v27n01_02

22 Would using an antecedent rubric that provided immediate feedback work with STUDENTS?

23

24 New rubrics default to three rows and three columns. Access Course Tools and select Rubrics. On the Rubrics page, click Create Rubric. Type a Name for the rubric. Click Submit.

25

26

27 Select the rubric

28

29 How to Associate a Rubric to a Discussion Board

30

31

32 Click on icon "View Rubric" (clicking on this icon will give you feedback on the metrics this assignment is being scored on).

33 How to Grade an assignment using the interactive Rubric

34

35

36

37

38 How to Grade a Discussion Board via Rubric

39

40

41

42

43

44

45 Rubric delivery may be instrumental in affecting student performance * Department studies website * * Spreadsheet imbedded into the assignment * My favorite paper * Interactive rubric

46 Average omissions = total number of omissions / number of submissions

47

48

49 Free Rubric Navigational Guide: Questions?