From: Academic Dean and Provost March 30, 2015 AY2015 ANNUAL ASSESSMENT REPORTS - FACULTY SENATE ASSESSMENT COMMITTEE

Transcription

1 From: Academic Dean and Provost March 30, 2015 Subj: AY2015 ANNUAL ASSESSMENT REPORTS - FACULTY SENATE ASSESSMENT COMMITTEE Ref: (a) ACDEANINST A of 8 Sep 14 Enc1: (1) AY2015 Department Assessment Report Template (2) AY2015 Division Assessment Report Template (3) Departmental Learning Outcomes and Curriculum Map Template (4) USNA Attributes and Departmental Learning Outcomes Map Template (5) USNA Core Learning Outcomes and Core Course Outcomes Map Template (6) AY2015 Faculty Senate Assessment Committee Report Rubric 1. Background As stated in ACDEANINST A of 8 Sep 14, the fundamental purpose of the Naval Academy academic assessment program is to support continuous improvement of midshipman learning and development. 2. AY2015 Department and/or Program Assessment Reports Assessment reports come in one of two forms: a retrospective report or an annual status report. Reports should be written so as to be understandable by members of the Faculty Senate Assessment Committee, in particular those outside the discipline creating the report, and should provide sufficient information about the assessment methods that report reviewers can understand the process used by the department or program. During most years departments and programs will submit an annual status report documenting assessment processes, decisions based on the results of previous assessments, and assessment activities undertaken in the most recent academic year as detailed in Encl (1). Annual status reports should include direct evidence of student learning, using student work products, and the programmatic decisions based on this evidence should be summarized in the body of the report. Reports should provide information about the processes and methods used to collect/observe and analyze student work products Supplementary materials such as assessment instruments, graphs and tables of results, and other relevant supporting information should be included as appendices. A retrospective report is an option when the program is preparing for a periodic, program specific external review (such as a USNA visiting committee (when assessment of student learning is a large component of the preparation), ABET, or ACS review), or when the Naval Academy itself is preparing for a Middle States Periodic Review Report (PRR) or decennial review. Recognizing that all departments participate in periodic program-specific external reviews, departments are strongly encouraged to leverage the assessment instruments, data, and analysis utilized for these reviews and not to duplicate their efforts unnecessarily. All reports, retrospective and annual status, are expected to include the departmental student learning outcomes maps to the curriculum and departmental student outcomes to the USNA attributes.

2 3. AY2015 Division Assessment Reports Each division is required to submit an annual assessment report. The AY2015 Division Core Assessment Reports should describe the Division Core Assessment Status regarding the implementation of divisions assessment plans. A template is provided in Enc1 (2). 4. Action In keeping with the ACDEANINST A dated 8 Sep 14 those responsible for assessment at the Division and Department levels including those responsible for interdisciplinary programs will prepare assessment reports for AY2015 and submit them electronically to the Faculty Senate Assessment Committee through the ex-officio chair, Dr. Katherine Cermak, Assistant Dean for Planning and Assessment, at the end of the 2015 spring semester, but no later than 15 June Reports will be shared with the Academy Effectiveness Board s committee on institutional effectiveness to use in their deliberations. Reports are also used to document assessment activities for accreditation purposes and internally to inform other areas of the state of assessment at USNA, good practices and possible areas of collaboration. Distribution: Chair, Faculty Senate Assessment Committee Division Directors Senior Professors Departmental/Program Chairs Departmental/Program Assessment Coordinators A. T. PHILLIPS

3 AY2015 Department / Program Annual Status Assessment Report Template The AY2015 Department and/or Program Annual Status Assessment Report should include the following information: I. Follow-Up on Last Year s Assessment Report Recommendations 1. Actions taken based on last year's (AY2014) assessment activities and results, where available. Include relevant supporting data as an appendix. II. III. Report on Current Academic Year s Assessment Activities 1. Assessment(s) implemented this academic year: i. Indicate the departmental or program student learning outcome(s) assessed. ii. Provide a description of the assessment sources, including student work, and methods used by the department and/or program to collect data. How was student work collected or observed? What student group(s) were studied? How were they selected? iii. Include a description of the process for organizing and analyzing the assessment data. What methods were used to analyze and interpret the results? What person or group analyzed the student work? Please include rubrics, scoring guides, etc. as an appendix. 2. Results obtained, including an indication of the degree to which students met the expectations of learning related to the outcome(s), as determined by the department or program. 3. Lessons learned and actions recommended based on this year s (AY2015) assessment activities. What are the suggested curricular and/or assessment changes? How will this information be shared with appropriate groups (department/division/etc.)? An overview of the program assessment plans for each program coordinated/administered by the department*. 1. It is expected that this overview will remain largely the same from year to year. However, revisions should be made to reflect changes in the curriculum or the assessment plan and highlighted in the annual report. 2. This overview should: i. Include a list of student learning outcomes. ii. Include maps of the outcomes to the curriculum and institutional goals (see enclosures 3, 4, and 5). iii. Describe the overall structure and operation of the assessment plan to assess all outcomes over a reasonable period (3-4 years). iv. Describe the current status with regard to the a ssessment time cycle and provide the assessment plans that are in place for AY * For departments that have oversight responsibility for more than one program (i.e., have 2 or more majors/minors/tracks), please provide a summary of the assessment of each one. This includes, but is not limited to General Science, General Engineering, and Quantitative Economics. Direct evidence of student learning and the programmatic decisions based on this evidence should be summarized in the body of the report. However, supplementary materials such as assessment (Enclosure 1)

4 instruments, graphs and tables of results, and other relevant supporting information should be included as appendices. Note that core-related assessment results should be submitted for inclusion in the Division report. Reports should be written so as to be understandable by members of the Faculty Senate Assessment Committee, in particular those outside the area being reported, and should provide sufficient information about the assessment methods that report reviewers can understand the process used by the department or program. Supplementary materials such as assessment instruments, and graphs and tables of results should be included as appendices. (Enclosure 1)

5 AY2015 Division Assessment Report Template The AY2015 Division Assessment Report should include the following: I. A description of the plan to assess Division learning outcomes across departments within the division and across divisions, as appropriate, and a discussion of sources of assessment data. 1. It is expected that this overview will remain largely the same from year to year, and revisions will be made as needed to reflect changes in the Division Assessment Plan. i. List the divisional student learning outcomes to be achieved by graduation. (Include any updates of the Division Learning Goals.) ii. Describe the overall structure and operation of the Division Assessment Plan to assess all outcomes over a reasonable period (3-4 years). iii. Describe the current status with regard to the Division Assessment Time Cycle. II. III. IV. Actions taken based on last year's (AY2014) assessment activities. Include all relevant supporting data as an appendix. Lessons learned and recommendations for action based on this year's (AY2015) assessment activities including: 1. Assessment(s) implemented this academic year. (Include a summary of the assessment sources and methods used by the Division to collect data and the process for organizing and analyzing the assessment data.) a. Indicate which learning outcomes were assessed and if they were observable/measurable. b. Describe how student work was collected or student efforts were observed for the assessment. c. Indicate the department s (or the division s) minimum expectations of student performance for the outcome(s) the extent to which students meet those expectations. 2. Indicate how the core course learning outcomes align with the divisional outcomes and the seven attributes of a USNA graduate. 3. Describe plans to improve student learning, the curriculum, or assessment, as appropriate given the results. Division level review of the effectiveness of the core curriculum. 1. Indicate to what extent inter/intra division coordination, discussion, and plans for future activities related to the core are taking place or planned. 2. Indicate the overall effectiveness of the core (and its constituent courses). 3. Explain how the divisional assessment committee will provide feedback to the department/program chairs, faculty, and committees responsible for assessment of the core curriculum. Direct evidence of student learning and the programmatic decisions based on this evidence should be summarized in the body of the report. However, supplementary materials such as assessment instruments, graphs and tables of results, and other relevant supporting information should be included as appendices. Reports should be written so as to be understandable by members of the Faculty Senate Assessment Committee, in particular those outside the area being reported, and should provide sufficient information about the assessment methods that report reviewers can understand the process used by the department or (Enclosure 2)

6 program. Supplementary materials such as assessment instruments, and graphs and tables of results should be included as appendices. (Enclosure 2)

7 DEPARTMENT LEARNING OUTCOMES AND CURRICULUM MAP DEPARTMENT: RESPONDENT/DATE: List your Department Learning Outcomes across the top of the matrix and the Courses in your Curriculum where these outcomes are addressed in the first column. Where appropriate indicate the level of development within the cell the courses in which a Learning Outcome is Introduced (I), Reinforced (R), and Mastered (M) Assessed (A).* DEPARTMENT LEARNING OUTCOMES COURSES * If introduced/reinforced/mastered/assessed is not appropriate for you discipline please indicate the levels of development that are appropriate (e.g., low/medium/high or basic/intermediate/advanced, etc). (Enclosure 3)

8 Examples: COURSES/OUTCOMES 1. APPLY PRINCIPLES OF CHOREOGRAPHY 2.COMMUNICATE MEANING IN DANCE 3. WORK WITH PARTNERS TO DANCE MOVEMENT AND TEAMS Dance 101 (I) (I) Dance 201 (R) (I/R/A) (I) Dance 301 (M/A) (R/M/A) (R/A) COURSES/OUTCOMES 1. APPLY PRINCIPLES OF CHOREOGRAPHY TO DANCE 2.COMMUNICATE MEANING IN DANCE 3. WORK WITH PARTNERS AND TEAMS MOVEMENT Dance 101 (I) Incorporate simple dance forms and demonstrate principles of balance and opposing weight. (I) Interpret a range of emotions and expressions expressed through gesture, posture, and movement. Dance 201 (R) Coordinate movement with a variety of rhythms and styles. (I/R/A) Use gestures, posture, and movement to convey emotion and expression. (I) Collaborate with a partners and groups (e.g., echoing, leading/following, mirroring, calling/responding, sequence construction) Dance 301 (M/A) Use dance elements to create short dance pieces with various rhythms, styles, and dynamics. (R/M/A) Perform original pieces that effectively communicate artistic intent. (R/A) Use partner/group relationships to define spatial floor patterns and shape designs. (Enclosure 3)

9 DIVISION OR DEPARTMENT LEARNING OUTCOMES AND USNA ATTRIBUTES MAP DIVISION OR DEPARTMENT: RESPONDENT/DATE: List your Division or Department Learning Outcomes across the top of the matrix. Where appropriate indicate in the cells the Courses in your Curriculum that demonstrably address the Outcome and the Attribute. Additionally, please and indicate the extent to which the course addresses the Attribute: Partially (P) (some, but not all of an Attribute is demonstrably addressed) or Completely (C) (all of an Attribute is demonstrably addressed). DIVISION OR DEPARTMENTAL LEARNING OUTCOMES ATTRIBUTES* Selfless 2. Inspirational 3. Proficient 4. Innovative 5. Articulate 6. Adaptable 7. Professional *USNA Midshipmen Attributes 1. Selfless leaders who value diversity and create an ethical command climate through their example of personal integrity and moral courage. 2. Mentally resilient and physically fit officers who inspire their team to accomplish the most challenging missions and are prepared to lead in combat. 3. Technically and academically proficient professionals with a commitment to continual learning. 4. Critical thinkers and creative decision makers with a bias for action. 5. Effective communicators. 6. Adaptable individuals who understand and appreciate global and cross-cultural dynamics. 7. Role models dedicated to the profession of arms, the traditions and values of the Naval Service and the constitutional foundation of the United States. (Enclosure 4)

10 EXAMPLE: List your Division or Department Learning Outcomes across the top of the matrix. Where appropriate indicate in the cells the Courses in your Curriculum that demonstrably address the Outcome and the Attribute. Additionally, please and indicate the extent to which the course addresses the Attribute: Partially (P) (some, but not all of an Attribute is demonstrably addressed) or Completely (C) (all of an Attribute is demonstrably addressed). ATTRIBUTES* CHOREOGRAPHY LEARNING OUTCOMES 1. APPLY PRINCIPLES OF 2.COMMUNICATE MEANING IN CHOREOGRAPHY TO DANCE DANCE MOVEMENT 3. WORK WITH PARTNERS AND TEAMS 1. Selfless P (Value diversity) (R) 2. Inspirational P (Physically Fit) (M) P (Inspire their team) (R) 3. Proficient 4. Innovative C (M) C (M) C (R) 5. Articulate P (M) 6. Adaptable P (Understand/Appreciate Global and Cross Cultural Dynamics) (M) 7. Professional (Enclosure 4)

11 CORE LEARNING OUTCOMES (CLOS) AND CORE COURSE OUTCOMES DEPARTMENT/CORE COURSE(S): RESPONDENT/DATE: List the Learning Outcomes for each Core Course (if courses have different outcomes it may be simpler to duplicate this table for each course) across the top of the matrix. Where appropriate indicate in the cells the Core Course Learning Outcomes that aligns with the CLO. Additionally, please and indicate the extent to which the CLO is addressed: Partially (P) (some, but not all of the CLO is explicitly addressed) or Completely (C) (all of a CLO is explicitly addressed). CORE COURSE LEARNING OUTCOMES CLOS* Apply leadership skills 2. Reason morally/ethically 3.Apply principles of naval science & the profession of arms 4. Solve technical problems 5. Communicate effectively 6. Critically reason 7. Understand American heritage (Enclosure 5)

12 8.Interpret past and current world events 9.Demonstrate intellectual curiosity *USNA Core Learning Outcomes USNA Core Learning Outcomes 1) Apply leadership skills: use contemporary leadership theories to support and motivate diverse team members in the accomplishment of objectives. 2) Reason morally/ethically: recognize moral dilemmas and use ethical frameworks and principles to generate solutions that embody the highest moral standards. 3) Apply principles of naval science and the profession of arms: Operate naval technology, demonstrate navigation skills, and generate solutions to given warfare scenarios. 4) Solve technical problems: apply fundamental principles from science, engineering, and mathematics to solve technical problems in both standard operating and unfamiliar contexts. 5) Communicate effectively: develop, organize, and communicate information and ideas through written, oral, and visual media. 6) Critically reason: sufficiently obtain, critically analyze, appropriately interpret, and use quantitative data and qualitative information to construct creative solutions to complex problems. 7) Understand American heritage: articulate core American values and diverse experiences with an awareness of multiple cultural contexts. 8) Interpret past and current world events: interpret past and current world events through an awareness of different cultures. 9) Demonstrate intellectual curiosity: apply self-directed learning strategies to improve knowledge, skills, and abilities beyond requirements. (Enclosure 5)

13 Example: List the Learning Outcomes for each Core Course (if courses have different outcomes it may be simpler to duplicate this table for each course) across the top of the matrix. Where appropriate indicate in the cells the Core Course Learning Outcomes that aligns with the CLO. Additionally, please and indicate the extent to which the CLO is addressed: Partially (P) (some, but not all of the CLO is explicitly addressed) or Completely (C) (all of a CLO is explicitly addressed). CHOREOGRAPHY CORE COURSE LEARNING OUTCOMES: DANCE 101/201/301 CORE LEARNING OUTCOMES 1. APPLY PRINCIPLES OF CHOREOGRAPHY TO DANCE MOVEMENT 2.COMMUNICATE MEANING IN DANCE 3. WORK WITH PARTNERS AND TEAMS 1. Apply leadership skills 201/301 (P): use contemporary leadership theories to support and motivate diverse team members in the accomplishment of objectives. 2. Reason morally/ethically 3.Apply principles of naval science & the profession of arms 4. Solve technical problems 5. Communicate effectively 101/201/301 (P): develop, organize, and communicate information and ideas through written, oral, and visual media. 6. Critically reason 7. Understand American heritage 8.Interpret past and current world events 9.Demonstrate intellectual curiosity (Enclosure 5)

14 Assessment Rubric Major/Core: For each criterion, please select the column that best describes the department s, program s or core courses overall current status. Exemplary Fully Met Developing Not Present Student Learning Outcomes (sometimes called objectives or goals): Assessment is an on-going process Learning outcomes represent what students will know or be able to do not only at the end of the program or core course(s), but how learning has progressed across courses/the program. Learning outcomes culminate in the highest (appropriate) levels of thinking. ( Dean/Assessment/index.php) Assessment is an on-going process of data collection, evaluation, and improvement. As appropriate, action(s) have been taken, based on the findings from past assessments. There is an intent to assess the effect of changes on student learning and outcome achievement. The loop has been closed with the intention to reexamine in a reasonable period of time. Learning outcomes are student centered statements of what students will know or be able to do. Learning outcomes represent what students will know or be able to do under certain conditions or at the end of the program or core course(s) Learning outcomes primarily focus on higher order thinking skills (apply, analyze, synthesize, and evaluate and not solely on knowing and understanding). The number of outcomes is reasonable to cover essential learning of the program and be assessable once approximately every 3 to 4 years. There are not too many outcomes (suggesting that some can be combined) or too few (suggesting that outcomes need to be unpacked). Assessment is an on-going process of data collection, evaluation, and improvement with outcomes scheduled to be assessed over a reasonable period of time (about every 3 to 4 years). Assessment is an on-going process of data collection, evaluation, and improvement. As appropriate, action(s) have been taken based on the findings from past assessments. The loop has been closed. Learning outcomes are not student centered instead indicating what the department or instructors will do. Learning outcomes exist, but do not indicate when students are expected to be able to demonstrate the knowledge or skill expected. Learning outcomes primarily focus on what students will know or understand, but not how they will use that knowledge or understanding. The outcomes either fail to cover essential learning within the program or address areas tangential to the program. Not all outcomes are scheduled to be assessed or assessment is occurring on an episodic basis. Assessment takes place, but there is no documentation that results are used to inform departmental discussions or decisions about teaching strategies or student learning. (Enclosure 6)

15 Alignment of Department/Program Learning Outcomes (Map) Alignment of Core Learning Outcomes (Map) Methodology Progressive development for each department/program learning outcome is identified throughout the department's curriculum. (Movement from introduced, to reinforced/ practiced, to where mastery/competency is achieved and where assessment takes place.) Multiple assessments including direct assessments of student learning are being used that clearly align with the outcome being addressed. Evaluation and analysis of student work is shared by multiple faculty members and, when appropriate, procedures for improving rater agreement (inter rater reliability) are indicated. Levels of development for each department/program learning outcome are identified within the department's curriculum. (Where learning is introduced, reinforced or practiced, where mastery or competency is achieved.) The portion of the Graduate Attribute that is addressed by the department/program learning outcome (including the complete Attribute when appropriate) is identified. The portion of the Core Learning Outcome that is addressed by the department/program learning outcome is identified. Direct assessment in which student work (essays, items on exams, assignments, presentations, etc.) has been selected that is appropriate for the outcome being measured. Student work either is an appropriately collected sample (simple random or systematic), a population, or otherwise appropriately selected to ensure that results are representative and the amount of work is feasible for the committee. More than one faculty member is involved in the evaluation and analysis of student work. Presence or absence of outcome identified for each course in the curriculum or levels are indicated only for the department's required courses. Presence or absence of outcome identified for each of the USNA's Seven Attributes of Graduates. Presence or absence of outcome identified for each of the Core Learning Outcomes. Indirect assessments are solely being used that do not directly examine student work. Possibilities include student perceptions of ability, grades not specifically and solely linked to outcomes, faculty evaluations that do not link to student work. Assessment materials do not include student work and/or are gathered on a volunteer or an ad hoc basis. The collected materials are either too little or too much for the committee to reasonably examine. Assessment of student work or other assessment materials takes place in isolation and/or analysis of results is primarily handled by a single individual. Assessment tools (rubrics, exam items, scoring guides, etc.) and methods align with the learning outcome being assessed. Assessment tools (rubrics, exam items, scoring guides, etc.) and methods do not align with the outcomes being assessed. Performance Target level of performance is indicated for a student at various points within the program reflecting expected development. Target level of performance is indicated for a student at the end of the program and appears appropriate. Criteria for different levels of performance have been indicated, but expectations are not clearly identified or are inappropriate (much too high or too low). (Enclosure 6)

16 (Enclosure 6)