ABET & NCAAA ACCREDITATION Course Design Workshop

Transcription

1 ABET & NCAAA ACCREDITATION Prof. Ali M. Al-Bahi Director of the Academic Accreditation Unit Faculty of Engineering, King Abdulaziz University Modified and Presented by Dr. Faisal Iskanderani UBTVR for Q&D Sept 11, 2014

2 Session #1 Introduction why Learning Outcomes? Student Learning Experience September 11,

3 Introduction September 11,

4 Introduction CEIT will be working towards National, and International Accreditation ( NCAAA and ABET) Time line: end of 2016 Today s workshop will deal with one component of accreditation, i.e., Students Learning Outcomes delay this

5 Introduction ABET and NCAAA accreditation is assurance that a college or university program meets the quality standards established by the profession for which it prepares its students. NCAAA and ABET have similar accreditation criteria We will work together in parallel to achieve both at the same time, using the same documentation

6 ABET Documents Criteria for Accrediting Engineering Programs Accreditation Policy and Procedure Manual Template for the Engineering Self Study Report Course binders or files NB: Documents should be downloaded for the year of the accreditation cycle from: September 11,

7 NCAAA Documents National Qualifications Framework, NCAAA Standards for the Accreditation and Quality Assurance for Institutions and Programs (11 Standards), Self Evaluation Scales for NCAAA Standards, and Forms and Templates for Institutions and Programs as follows:

8 NCAAA Forms and Templates Course Specification Course Report Field Experience Specification Field Experience Report Program Specification Annual Program Report Template for Report on Periodic Program Self-Study

9 Accreditation Time-Line (a 2-3 year process) Note: In addition to one year for preparations September 11,

10 ABET Criteria for Accrediting Eng. Programs 1. Students 2. Program Education objectives (PEOs). 3. Student Outcomes (SOs). We will call them Program Learning Outcomes 4. Continuous Improvement 5. Professional Component (curriculum). 6. Faculty. 7. Facilities. 8. Institutional Support. + Program Criteria. September 11,

11 ABET Criteria for Accrediting Eng. Programs 1. Students 2. Program Education objectives (PEOs). 3. Student Outcomes (SOs). We will call them Program Learning Outcomes 4. Continuous Improvement 5. Professional Component (curriculum). 6. Faculty. 7. Facilities. 8. Institutional Support. + Program Criteria. September 11,

12 NCAAA Standards for Accreditation and Quality Assurance 1. Mission and objectives (ABET C2) 2. Governance and administration (ABET C6 ) 3. Management of quality assurance and improvement (ABET C4) 4. Learning and teaching (ABET C3) 5. Student administration and support services (ABET C1) 6. Learning resources (Part of ABET C7) 7. Facilities and Equipment (Part of ABET C7) 8. Financial planning and management (ABET C8) 9. Employment processes (Part of ABET C6) 10. Research (Part of ABET C6) 11. Institutional relationships with the community (ABET C2)

13 NCAAA Standards for Accreditation and Quality Assurance 1. Mission and objectives (ABET C2) 2. Governance and administration (ABET C6 ) 3. Management of quality assurance and improvement (ABET C4) 4. Learning and teaching (ABET C3) 5. Student administration and support services (ABET C1) 6. Learning resources (Part of ABET C7) 7. Facilities and Equipment (Part of ABET C7) 8. Financial planning and management (ABET C8) 9. Employment processes (Part of ABET C6) 10. Research (Part of ABET C6) 11. Institutional relationships with the community (ABET C2)

14 NCAAA Domains of Learning Outcomes knowledge, the ability to recall, understand, and present information including: 1. knowledge of specific facts, 2. knowledge of concepts, principles and theories, and 3. knowledge of procedures. cognitive skills, the ability to apply understanding of concepts, principles, theories and procedures in critical thinking and creative problem solving, both when asked to do so and when faced with unanticipated new situations. September 11,

15 NCAAA Domains of Learning Outcomes Interpersonal skills and responsibility, the ability to: 1. take responsibility for their own learning and continuing personal and professional development, 2. work effectively in groups and exercise leadership when appropriate, 3. act responsibly in personal and professional relationships, and 4. act ethically and consistently with high moral standards in personal and public forums. Communication, information technology and numerical skills, including the ability to: 1. communicate effectively in oral and written form, 2. use information and communications technology, and 3. use basic mathematical and statistical techniques. Psychomotor skills involving manual dexterity (this domain applies only to some programs). September 11,

16 NCAAA Domains of Learning Versus ABET Outcomes Domain of Knowledge (Not existing in ABET): 1. Ability to recall, understand, and present knowledge of facts, concepts, principles and theories of Math and basic sciences. 2. Ability to recall, understand, and present knowledge of discipline-related practices and procedures. Domain of cognitive skills (corresponding to a, b, c, e, h, & j) 3. Ability to identify, formulate and solve real-life discipline related problems. 4. Ability to design, implement and evaluate a discipline related system or process to satisfy desired needs. 5. Ability to identify, analyze, and critically evaluate arguments and take informed criteria-based decisions.

17 NCAAA Domains of Learning Versus ABET Outcomes Domain of Interpersonal skills (corresponding to i, d, & f) 6. Ability to take responsibility for their own learning and continuing personal and professional development 7. Ability to work effectively in groups and exercise leadership when appropriate. 8. Ability to act responsibly in personal and professional relationships. 9. Ability to act ethically and consistently with high moral standards in personal and public forums.

18 NCAAA Domains of Learning Versus ABET Outcomes Domain of Communication, IT, and numerical skills (corresponding to a, g, & k) 10. Ability to communicate effectively in oral and written forms 11. Ability to use information and communications technology. 12. Ability to use basic mathematical and statistical techniques. Domain of psychomotor skills (not existing for engineering, IT, or managerial education)

19 Why Learning Outcomes? September 11,

20 Why Learning Outcomes? ABET Definition of Program Learning Outcomes Statements that describe what the students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills, and behaviors that the students acquire as they progress through the program. September 11,

21 Why learning outcomes? Course Learning Outcomes Each course should have a set of specific learning outcomes that lead to: More measurable student outcomes Better assessment Higher quality feedback Improved courses and programs Improved student learning and achievement September 11,

22 Infusion of Learning Outcomes into Courses Design Backward Intended Learning Outcomes of the Lesson Intended Learning Outcomes of the Unit Intended Learning Outcomes of the Course Program Outcomes September 11, 2014 Deliver Forward 21

23 Criterion 3 Program Learning Outcomes (a-k) a)an ability to apply knowledge of mathematics, science, and engineering, b)an ability to design and conduct experiments, as well as to analyze and interpret data, c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability, d)an ability to function on multi-disciplinary teams, e)an ability to identify, formulate, and solve engineering problems, September 11,

24 Criterion 3 Program Learning Outcomes (a-k) f) an understanding of professional and ethical responsibility, g) an ability to communicate effectively, h) the broad education necessary to understand the impact of engineering solutions in a global and societal context, i) a recognition of the need for, and an ability to engage in life-long learning, j) a knowledge of contemporary issues, and k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice. September 11,

25 NCAAA and ABET Definitions for Learning and Teaching Program Educational Objectives: Broad statements that describe what graduates are expected to attain within a few years after graduation. Program educational objectives are based on the needs of the program s constituencies. Program Learning Outcomes: Describe what students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills, and behaviors that students acquire as they progress through the program. September 11,

26 NCAAA and ABET Definitions for Learning and Teaching Course Learning Outcomes: Describe what students are expected to know and be able to do by the end of the course delivery. These relate to the knowledge, skills, and behaviors that students acquire as they progress through the course. September 11,

27 NCAAA and ABET Definitions for Learning and Teaching (Cont d) Assessment : one or more processes that identify, collect, and prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the outcome or objective being measured. Appropriate sampling methods may be used as part of an assessment process. September 11,

28 NCAAA and ABET Definitions for Learning and Teaching (Cont d) Evaluation: one or more processes for interpreting the data and evidence accumulated through assessment processes. Evaluation determines the extent to which student outcomes and program educational objectives are being attained. Evaluation results in decisions and actions regarding program improvement September 11,

29 Mission, Objectives, Outcomes & Learning Experience Undergraduate Studies Upon Graduation Work environment Mission Learning Experience Curriculum & Other Activities Student Outcomes of the Program Program Objectives Mission Accomplished September 11,

30 Mission, Objectives, Outcomes & Learning Experience Undergraduate Studies Upon Graduation Work environment Mission Learning Experience Curriculum & Other Activities Program Learning Outcomes Program Objectives Mission Accomplished September 11,

31 Mission, Objectives, Outcomes & Learning Experience Undergraduate Studies Upon Graduation Work environment Mission Learning Experience Curriculum & Other Activities Program Learning Outcomes Program Objectives Mission Accomplished September 11,

32 Mission, Objectives, Outcomes & Learning Experience Undergraduate Studies Upon Graduation Work environment Mission Learning Experience Curriculum & Other Activities Program Learning Outcomes Program Objectives Mission Accomplished September 11,

33 Student Learning Experience September 11,

34 Student Learning Experience A path to achieve Program Learning Outcomes and foster the attainment of program educational objectives: 1. Curriculum Design (into courses and other learning activities) 2. Course Design 3. Teaching and Learning 4. Assessment (course-level and program-level) 5. Evaluation and continuous improvement

35 Teaching versus Learning Education has become outcome-based instead of being content-based. Educational activities became centered around learning instead of teaching. INPUTS OUTCOMES INPUTS Learning OUTCOMES

36 Teaching versus Learning Education has become outcome-based instead of being content-based. Educational activities became centered around learning instead of teaching. INPUTS OUTCOMES INPUTS Learning OUTCOMES

37 Session # 2 Domains of Learning Levels of Learning Writing Course Learning Outcomes September 11,

38 Domains of Learning September 11,

39 Domains of the Learning Outcomes The outcomes of any learning activity, or the change in the learner s behavior, may fall in one of 3 distinguished domains: Cognitive Domain that involves knowledge and the development of intellectual skills Affective Domain that includes the manner in which we deal with things emotionally, such as feelings, values, appreciation, enthusiasms, motivations, and attitudes. Psychomotor Domain that includes physical movement, coordination, and use of the motor-skill areas. Development of these skills requires practice and is measured in terms of speed, precision, distance, procedures, or techniques in execution. September 11,

40 Levels of Learning September 11,

41 Cognitive Domain (mental) Levels of Learning Synthesis (Creative Thinking) Evaluation (Critical Thinking) Analysis Application Comprehension Knowledge September 11,

42 Classical Terms Evaluation Synthesis Analysis Application Comprehension Knowledge Levels of Learning New Terms Creating Evaluating Analyzing Applying Understanding Remembering (Based on Pohl, 2000, Learning to Think, Thinking to Learn, p. 8) The names of six major categories were changed from noun to verb forms. September 11,

43 Affective Domain (Attitudes) Degrees of internalization This domain includes the manner in which we deal with things emotionally, such as feelings, values, appreciation, enthusiasms, motivations, and attitudes. 1. Receiving: Awareness. Willingness to hear. Selected Action Verbs : chooses, asks, replies, selects 2. Responding: Active participation. Attends to and demonstrates compliance, willingness or satisfaction in responding. Selected Action Verbs: complies, conforms, assists 3. Valuing: Is based on the internalization of a set of values, and are expressed in overt and identifiable behavior. Selected Action Verbs: Demonstrates, proposes, initiates, justifies 4. Organizing: Values are organized into priorities. The emphasis is on comparing, relating, and synthesizing values. Selected Action Verbs: arranges, integrates, synthesizes, defends 5. Internalizing (characterization): A value system that controls behavior. Behavior is pervasive, consistent, predictable, and characteristic of the values held. Selected Action Verbs : acts, proposes, discriminates.

44 Psychomotor Domain (levels of physical skills) The psychomotor domain includes physical movement, coordination, and use of the motor-skill areas. Development of these skills requires practice and is measured in terms of speed, precision, distance, procedures, or techniques in execution. 1. Set: Readiness to act Includes mental, physical and emotional sets. Selected Action Verbs : begins, displays, moves, proceeds 2. Guided Response: Early stages in learning a complex skill. Involves imitation and trial & error. Selected Action Verbs: copies, traces, follows, reproduces, responds 3. Mechanism: Intermediate skill in learning a complex skill. Includes habituation and proficiency. Selected Action Verbs: assembles, constructs, dismantles, displays 4. Complex Overt Response: Skillful performance that involves complex movement patterns. Proficient in speed and accuracy. Selected Action Verbs: Same as Mechanism, but will have adverbs that indicate performance is quicker, more accurate and automatic 5. Adaptation: Skills are well-developed and the individual can modify to fit special requirements. Selected Action Verbs: adapts, alters, revises 6. Origination: Creating new patterns to react to particular situations. Emphasis on creativity based on highly developed skills. Selected Action Verbs: Builds, combines, composes.

45 A Fluid Mechanics Example Level Explanation Sample Question Remembering Understanding Applying The student is expected to recite memorized information about the concept. The student is expected to explain the concept in his or her own words. The student is expected to apply the concept to a particular situation. What is a fluid? Why mercury is considered as a fluid? Calculate the position of the shock wave in the given nozzle Analyzing Evaluating Creating The student is expected to separate materials or concepts into component parts so that their organizational structure may be understood. The student is expected to make judgments about the value of ideas or materials. The student is expected to put parts together to form a whole, with emphasis on creating a new meaning or structure. Analyse the flow field inside the test section a high speed wind tunnel. Compare the given two nozzle designs that perform the same task, which one is better and why? Design a variable area nozzle to execute the following flight mission.

46 A Computer Programming Example Level Explanation Sample Question Remembering Understanding Applying Analyzing Evaluating Creating The student is expected to recite memorized information about the concept. The student is expected to explain the concept in his or her own words. The student is expected to apply the concept to a particular situation. The student is expected to separate materials or concepts into component parts so that their organizational structure may be understood. The student is expected to make judgments about the value of ideas or materials. What is a program? How is a program similar to a recipe? What is the output of the following program Identify the customer high level requirements based on the following customer interview. What additional questions you may need? Given two programs that perform the same task, which one is better and why? The student is expected to put parts together to Write a program to perform a form a whole, with emphasis on creating a new given task meaning or structure.

47 Writing Learning Outcomes September 11,

48 Writing Learning Outcomes While it is possible to write learning outcomes of all types for each of the three domains, the vast majority are written for the cognitive domain.

49 Action Verbs (used in writing learning outcomes) Taxonomy Categories Remembering Sample Verbs for Stating Specific Learning Outcomes Cite, Define, Identify, Label, List, Match, Name, Recognize, Reproduce, Select, State Understanding Classify, Convert, Describe, Distinguish Between, Explain, Extend, Give Examples, Illustrate, Interpret, Paraphrase, Summarize, Translate Applying Apply, Arrange, Compute, Construct, Demonstrate, Discover, Modify, Operate, Predict, Prepare, Produce, Relate, Show, Solve, Use, calculate Analyzing Analyze, Associate, Determine, Diagram, Differentiate, Discriminate, Distinguish, Estimate, Infer, Order, Outline, Point-out, Separate, Subdivide, Model. Evaluating Appraise, Assess, Compare, Conclude, Contrast, Criticize, Discriminate, Evaluate, Judge, Justify, Support, Weigh Creating Combine, Compile, Compose, Construct, Create, Design, Develop, Devise, Formulate, Integrate, Modify, Organize, Plan, Propose, Rearrange, Reorganize, Revise, Rewrite, Tell, Write September 11,

50 NCAAA Suggested Guidelines for Learning Outcome Verbs Learning Domains Knowledge Cognitive Skills September 11, 2014 Meaning List of basic knowledge that students should know and understand when they complete the course. List of the thinking and problem solving skills the course is intended to develop. The list should include the ability to: (1) Use analytic and predictive formulae as well as conceptual tools that students should use when asked to do so. (2) Identify and use the tools that are appropriate for new and unanticipated problems. Suggested Verbs list, name, record, define, label, outline, state, describe, recall, memorize, reproduce, recognize, record, tell and write. estimate, explain, summarize, write, compare, contrast, diagram, subdivide, differentiate, criticize, calculate, analyze, compose, develop, create, prepare, reconstruct, reorganize, summarize, explain, predict, justify, rate, evaluate, plan, design, measure, judge, justify, interpret and appraise. 49

51 NCAAA Suggested Guidelines for Learning Outcome Verbs Learning Domains Meaning Suggested Verbs Interpersonal Skills & Responsibility List of the outcomes related to students interpersonal skills, capacity for self directed learning, and personal and social responsibility. demonstrate, judge, choose, illustrate, modify, show, use, appraise, evaluate, justify, analyze, question, and write. Communication, Information Technology, Numerical List of outcomes related to students communication, IT and numerical skills. demonstrate, calculate, illustrate, interpret, research, question, operate, appraise, evaluate, assess, and criticize. September 11,

52 1 st Break (10 Minutes) September 11,

53 Session # 3 Elements of Course Design Step by Step Course Design & Assessment

54 Elements of Course Design

55 To ABET Program Committee Elements of Course Design Modified from Felder and Brent Student outcomes Levels of Learning Instructor's goals Lectures Institutional technologies Labs Course Goals & Course Intended Learning Outcomes Students Exams and Quizzes Homework Feedback from ABET Program Committee Project Instruction Assessment Plus & Delta Syllabus Student expectations Learning Methods: Problem-Based, Project-based, Active Cooperative Student Portfolio Process Check Surveys

56 STEP BY STEP COURSE DESIGN & Assessment

57 STEP BY STEP COURSE DESIGN & Assessment 1. Determine Program Core Courses 2. Apply Course Design Methodology for each Core Course: Define the goals of the course Define the Course Intended Learning Outcomes Construct the Course Mapping Matrix Construct the Course Articulation Matrix Prepare the course syllabus 3. Construct Mapping Matrix for all core courses 4. Apply Course Assessment Methodology for each Core Course 5. Evaluate Course Assessment results

58 Program Core Courses Program core courses: a set of courses designated to address some or all of the Program outcomes. Compulsory courses in the Program would be obvious candidates for the core. Compulsory courses given by other Programs, such as math & physics might be included as long as they consistently address outcomes. Elective courses or courses whose content varies from one offering to another can't be included.

59 Step #1: Define the goal or goals of the course Two to three lines : The course is intended to. Example (Data Base Management Systems): Example: The course is intended to: Develop the student s ability to understand, design, and implement Data Base Management tools.

60 Step #2: Define The Course Intended Learning Outcomes Intended Learning Outcomes: specific observable (measurable) actions that the students should be able to perform if they mastered the course. i.e. by the completion of this course, the students should be able to: The first word must be one action verb from the process descriptive verbs related to LOL (see the first part of this workshop)

61 Step #2: Define The Course Intended Learning Outcomes Look at the items or concepts in detailed course content. For each item or concept decide what are the actions or tasks you want the students to do in order to demonstrate their level of mastery. Also think of how you will measure these actions (i.e. your assessment tools) and write down a suitable task statement. Each task statement includes one key action verb [such as list, explain, calculate, estimate, derive, model, design, choose, and criticize] along with a definition of the task and possibly a specification of the conditions under which the task is to be performed.

62 Step #2: Define The Course Intended Learning Outcomes Do you want the students to define terms, explain them, apply them to known situations, to new situations, or use them to design a new artifact or process? Think of the levels of learning and their associated action verbs and associate a targeted level of learning to each course learning outcome. The best choice is to have a maximum of 9 to 12 course learning outcomes (one to 2 per course topic). The more they are detailed, the easier you could design their assessment, but you may find it difficult to have sufficient assessment tools.

63 Step #2: Define The Course Intended LO (EXAMPLE) Course Title: Introduction to Database Systems Current Catalog Description: Introduction to the database management systems, database administration, data processing, data modeling, database design, development, and implementation. Contrasts alternative modeling approaches. Includes implementation of current DBMS tools and SQL. Major Topics Covered in the Course: Data modeling Logical database design Physical database design (including performance issues) SQL and other database access methods Database Administration (including Data Warehousing and Administration)

64 Step #2: Define The Course Intended Learning Outcomes (Good Examples) Data modeling 1. Model data relevant to a database task, given written description, reports and other information from a systems user. 2. Model data using entity-relationship as well as object-oriented methods. Logical database design 3. Transform entity-relationship model into a logical design, following the relational approach. Physical database design (including performance issues) 4. Assess aspects of the physical design, and transform the logical design accordingly.

65 Step #2: Define The Course Intended Learning Outcomes (Bad Examples) Understand the fundamentals of data base design (not measurable) Students demonstrate that they can derive and properly apply the equations of. (redundant words- what is meant by properly apply? In which situation) Statements of non-observable actions (such as learn, know, understand, or appreciate ) might qualify as educational objectives BUT NOT Intended learning Outcomes Understanding cannot be directly observed; the student must do something observable to demonstrate his understanding.

66 Session # 3 Hands-on Exercise Forming work teams Define the course goals Define a set of Intended course learning outcomes Presenting

67 Forming Teams September 11,

68 Forming work Teams Now we will form teams of 3 members each. All members of each team belong to one program. Each team will select one core course that falls in a domain of their common interest. September 11,

69 Define the Course Goals September 11,

70 Step#1: Define the goal or goals of the course (10 minutes) Use the catalogue description of the course to define its goals from the instructors view point Take 5 minutes to write 2-3 lines : The course is intended to. Remember the following example (Data Base Management Systems): The course is intended to: Develop the student s ability to understand, Design, and implement Data Base Management tools. September 11,

71 Define a set of Intended course learning outcomes September 11,

72 Step#2 Define the Course Intended Learning Outcomes (30 minutes) take 20 minutes to prepare 4 to 5 intended course learning outcomes (1 to 2 CLOs per major topic). Notice that each CLO should start by an action verb Define the level of learning for each CLO. Map each CLO to one or more of ABET Program Learning Outcomes. Follow the following examples: Physical database design: Assess aspects of the physical design, and transform the logical design accordingly. (Evaluating, level #3, outcome c). SQL and other database access methods Model data relevant to a database task, given written description from a systems user. (Analyzing, level #2, outcome c). September 11,

73 Empty Course Articulation Matrix Program LO September 11,

74 Presenting Your Work Now one team will come to podium. Take 5 minutes to present September 11,

75 Presenting September 11,

76 2 nd Break (30 Minutes) September 11,

77 Session # 4 Learning Depth and Breadth Apply Course Design Methodology for each Core Course Step #3. Construct the Course Mapping Matrix Step #4. Construct the Course Articulation Matrix Documenting Course Design Data September 11,

78 Learning Depth and Breadth September 11,

79 Learning Depth and Breadth 1. The Depth of Learning is measured by level of learning in the cognitive domain 2. The breadth of learning is measured by the number of contact hours allocated to the learning activities targeting an outcome. 3. The 6 levels of learning are grouped into 3 levels of learning depth: Level #1: low level of learning: Knowledge and Comprehension or Remembering and Understanding (weight = 1). Level #2: Medium level of learning: Application and Analysis or Applying and Analyzing (weight = 2). Level #3: High level of learning: Synthesis and Evaluation or Evaluating and Creating (weight = 3). September 11,

80 Student s Exposure 1. The product of depth of learning of a given student outcome by the corresponding breadth in a course indicates the student s exposure to this outcome in that course as compared to other courses addressing the same student outcome. 2. Student outcomes that have high exposure in a given course are considered as the critical outcomes of that course and the course is considered as a key course for these outcomes. 3. Add a column to your articulation matrix containing the CLO coverage hours. 4. Exposure of an outcome = SUM of (the Product of the alignment coefficient of the outcome with each CLO by CLO coverage). September 11,

81 Apply Course Design Methodology for each Core Course September 11,

82 STEP BY STEP COURSE DESIGN & Assessment 1. Determine Program Core Courses 2. Apply Course Design Methodology for each Core Course: Define the goals of the course Define the Course Intended Learning Outcomes Construct the Course Mapping Matrix Construct the Course Articulation Matrix Prepare the course syllabus 3. Construct Program Mapping Matrix for all core courses 4. Apply Course Assessment Methodology for each Core Course 5. Evaluate Course Assessment results September 11,

83 Step #3: Construct the course mapping matrix Course Learning Outcomes Program Learning Outcomes IE 301 LOL a b c d e f g h i j k l m n CLO_1 CLO_2 CLO_ Maximum Attainable : low levels of learning in Bloom s Taxonomy (Remembering and understanding) 2: medium levels of learning in Bloom s Taxonomy (Applying and Analyzing) 2: high levels of learning in Bloom s Taxonomy (evaluating and creating) September 11,

84 Step #4: Construct the course mapping matrix (Continued) Some schools fill the matrix with an indicator to demonstrate to which degree (High, medium, low) this course learning outcome (CLO) contributes to the achievement of the corresponding student outcome (using letters such as H, M, and L). Other schools indicate the corresponding level of learning (LOL). A simplest method is to use a binary system (ONE if the Course LO addresses the Program LO and ZERO if it does not). September 11,

85 Useful approach: Course Articulation Matrix Program LO September 11,

86 Empty Course Articulation Matrix Program LO September 11,

87 Documenting Course Design Data September 11,

88 Documenting Course Design Data A course articulation matrix assigns 1-2 Intended Course Learning Outcome (CLOs) to each topic and competency (a total of 9 to 12 CLOs per course). Each CLO is associated with a certain number of hours of exposure (contact hours allocated to this CLO). In-class & out-of-class learning activities are indicated for each CLO. Classical assessment tools are indicated for each CLO. CLOs are mapped into Student Learning Outcomes (a-k). Key outcomes are identified as those having the highest exposure.

89 Session # 4 Hands-on Exercise

90 Construct the Course Mapping Matrix (20 minutes) September 11,

91 Empty Course Articulation Matrix Program LO September 11,

92 Session # 5 Course Assessment & Course Binder

93 Course Assessment

94 Why Assessment Helps students understand where they should focus their time & energies Motivates students to do their best Helps students understand their strengths and weaknesses through feedback Gives students documentation of what they have learned which can be used to apply for jobs, awards, programs of advanced study, etc.

95 What Is Assessment? ABET Definition One or more processes that identify, collect, and prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the outcome or objective being measured. Appropriate sampling methods may be used as part of an assessment process.

96 Embedding Assessment With Regular Reflection Begin Section 1 Course Section 1 Teaching & Learning Assessment of Section 1 Topics Course Section 2 Teaching & Learning Begin Section 2 Evaluation of Achievement of Section 1 CLOs and Related SLOs Using HI-CLASS Revisions of Course Plan Embedding Assessment with Regular Reflection Approach Normal teaching Approach

97 Direct Assessment Methods (Clear and Compelling Evidence) Capstone experiences (research projects, presentations, theses, oral defenses, exhibitions, or performances) scored using a rubric Other written work, performances, or presentations, scored using rubrics Portfolios of student work Scores on locally-designed multiple choice and/or essay tests (e.g. final examinations in key courses, qualifying exams, comprehensive exams) accompanied by test blueprints describing what the tests assess Score gains between entry and exit on published or local tests or writing samples Summaries/analyses of electronic discussion threads Student reflections on their values, attitudes and beliefs, if developing those are intended outcomes of the course

98 Indirect Assessment Methods (Signs that Students Are Probably Learning, But Exactly What or How Much They Are Learning is Less Clear) Course grades (see next slide) Assignment grades, if not accompanied by a rubric or scoring guide (See an Example) Student ratings of their knowledge and skills and reflections on what they have learned in the course or program Questions on end-of-course student evaluation forms that ask about the course rather than the instructor

99 Assessment and Grading They do have one common characteristic: both intend to identify what the students have learned. Grades alone do not always give direct evidence: they don t identify which specific student learning outcomes and at what levels students have learned. Some course grades include additional student behaviors that are not related to student learning outcomes (e.g. attendance and participation). For example, a grade of B in a mathematics course indicates a student learned an acceptable level of math, but not the specific math knowledge or skills that the student mastered well or what areas need improvement. A project or assignment grade would be a better link to the evidence of student learning, but a rubric for the project or assignment would provide the best direct evidence and the level at which the student learned. Grades are based on direct evidence of student learning such as the evaluations of tests, papers, and projects, but need to be clearly linked and aligned to learning outcomes and/or rubrics to suffice as direct evidence for assessment purposes. September 11,

100 Proposed Course Level Assessment Tools Each semester Indirect measures: Course Learning Outcomes (CLOs) questionnaire (exit and sometimes entry) Course in-class and out-of-class learning activities questionnaire (exit) Student Outcomes (SOs) questionnaire (exit) Direct measures: Course Exams, HWs, Design projects, Written reports, Oral presentations, Course portfolios, Lab Reports evaluated using rubrics or mapped into CLOs Assignments for key SOs evaluated using rubrics.

101 Course Level Indirect Assessment Course Learning Outcomes (CLOs) Questionnaire: Put your course learning outcomes in the form of a questionnaire to measure the students confidence level in mastering the course learning outcomes. Use the 1 to 5 Eckert scale to cover Excellent-Medium-to-Poor mastery levels. Add some questions to evaluate the importance of course prerequisites. Distribute the questionnaire twice during the semester: in the 2 nd week (entry, if you admit this approach) and/or in the 12 th week (exit) of the course. Answer the questionnaire yourself to indicate your estimation of the students average level of mastery. (see an example)

102 Course Level Indirect Assessment (continued) Course learning activities (or instructional tools) Questionnaire: Ask the students to evaluate the impact of different inclass and out-of-class learning activities on their level of mastery of course learning outcomes. Use the 1 to 5 Eckert scale to cover Excellent-Medium-to-Poor impact. Distribute the questionnaire in 12 th week. Answer the questionnaire yourself to indicate your estimation of the impact of the learning activities. (see an example)

103 Course Level Indirect Assessment (continued) Student Outcomes Questionnaire: ABET Committee should divide each outcome into outcome elements and to define outcome attributes (measures) or KPIs associated with these outcome elements. A questionnaire should be developed for each outcome and used by the instructor who pretends that his course addresses this outcome. The questionnaire is to be distributed during the 12 th week of the semester. (see an example of outcome attributes/indicators and outcome Questionnaire)

104 Course Level Direct Assessment Direct Assessment of: HI-CLASS or Similar Assessment of Course Learning Outcomes (CLOs) Direct Approach Direct Rubric-Based Assessment of Key Student Outcomes.

105 Construct the Articulation Matrix (15 minutes) Now let us fill the Learning Activities and Assessment tools into the Matrix

106

107 THANK YOU

108 ABET & NCAAA ACCREDITATION Prof. Ali M. Al-Bahi Director of the Academic Accreditation Unit Faculty of Engineering, King Abdulaziz University Modified and presented by Dr. Faisal Iskanderani UBTVR for Q&D Sept 11, 2014 September 11,