Assessment Planning Guide for University of Rochester Graduate Degree Programs

Assessment Planning Guide for University of Rochester Graduate Degree Programs I. Introduction II. Useful outcomes-based program assessment: Designing an assessment plan that addresses key questions and concerns III. Steps in design of a program assessment plan Appendices 1. University of Rochester Doctoral Degree Program Assessment Plan Sample 2. University of Rochester Master s Degree Program Assessment Plan Sample 3. University of Rochester Doctoral or Master s Degree Program Assessment Plan Template 4. Sample dissertation proposal review rubric (all majors) 5. Sample dissertation defense review rubric (all majors) 6. Sample Master s thesis review rubric (all majors) 7. Princeton Univ. Biology PhD program dissertation review rubric 8. Univ. Virginia engineering and science dissertation defense review rubric 9. Examples of expanded program learning outcome statements for professional development, effective researcher, critical thinking, close reading of texts, writing Questions and support in writing a graduate program assessment plan: Barbara Masi, Director of Assessment Barbara.masi@rochester.edu 1

I. Introduction This document will introduce University of Rochester Arts, Sciences, and Engineering graduate degree programs to the terminology and methods for creating program assessment plans (sections I III). It also provides programs with useful templates and examples that can be used in creating plans (Appendices 1 9). II. Outcomes-based program assessment: Designing an assessment plan that addresses key questions and concerns Outcomes-based program assessment can add informational value to programs by formally documenting the achievements of faculty and students. It is also provides a more formal process for gathering student, alumni and faculty input for pointing out areas for program improvement. It is useful to take time as a faculty to discuss and clarify potentially perennial questions or concerns regarding graduate degree programs, and design an assessment plan that captures data to address them. For example, there may be concerns related to graduate students time to degree, preparation as productive and creative scholars, preparation for professions beyond the academy, or abilities related to teaching- all important goals of any graduate degree program. Since each degree program is unique, informal information shared among faculty might suggest key program factors that hinder student development in these areas. As part of a program s assessment plan, one can develop a plan with goals and assessment methods that uniquely incorporate this informal information and turn it into a formal data gathering process. Hence, assessment done well can achieve its desired goal of improving the quality of graduate education. III. Steps in design of a program assessment plan There are 9 steps in the design of a program assessment plan: 1. Define program objectives 2. Align objectives with curricular and other program activities in which objective addressed 3. Define program outcomes that permit measurement of each objective 4. Choose direct and indirect assessment methods that best measure student learning outcome 5. Design each chosen direct and indirect assessment method to ensure measurement quality 6. Set criterion / benchmark for success for each assessment method 7. Determine an implementation plan for each assessment method 8. Determine plan for reporting and review of assessment data, and how best to connect review with curricular change 9. Pull it all together in a complete program assessment plan using a plan template 2

1. Define Program Objectives Program objectives are the overall goals that define an effective graduate degree program. They often describe what a program is preparing students to do after graduation. Some common objectives of graduate degree programs include: Program will prepare students to contribute as original and creative scholars in their field Program will prepare students to as successful professionals in their field Program will prepare students to hold positions of leadership in academic, government, non-profit and industry organizations Program will prepare students as effective researchers in their field Program will prepare students as effective teachers 2. Align program objectives with curriculum and other academic activities (optional step) Curricular activities are designed to achieve program learning objectives. These activities include academic course work as well as activities that develop graduate students as professionals in the field. Some common activities of graduate programs include: Graduate courses in core disciplinary theory and methods Concentration/ specialization courses Research (other than thesis research) Thesis research Creative works Graduate seminar presentations, performances and other scholarly presentations that are organized by the program External conference presentations, journal publications, or performances Teaching assistant Professional society activity Internship 3. Define program learning outcomes Program learning outcomes are specific statements of program expectations for student abilities. They are both measurable and observable in student work or behavior and define how the program will achieve its program objectives. For example, a program might choose the following program objective and related program learning outcomes: Program objective: Program will prepare students as effective researchers in their field. Some related program learning outcomes are: a. read and review the literature in an area of study in such a way that reveals a comprehensive understanding of the literature 3

b. identify research questions/problems that are pertinent to a field of study and provide a focus for making a significant contribution to the field c. gather, organize, analyze, and report data using a conceptual framework appropriate to the research question and the field of study d. interpret research results in a way that adds to the understanding of the field of study and relates the findings to teaching and learning in science There are a number of commonly shared categories for master s and doctoral degree program learning outcomes. Some common categories, with program learning outcome examples, are (additional examples in Appendix 9): Core knowledge- ex. demonstrate comprehensive, in depth knowledge of the theories, methods, and scholarship in the field Specialization knowledge- ex. Demonstrate comprehensive, in depth knowledge of the theories, methods and scholarship within specialized area(s) including Creative synthesis- ex. creatively synthesize broad areas of theory and scholarship in generation of new ideas or insights or in critical analysis of works in the field Research- ex. conduct skillful research including gathering, processing, interpreting scholarship Methods- ex. make considered choices of and precisely implement methods of analysis or investigation as part of research projects Critical evaluation- ex. critically evaluate scholarly works in the field Written communication- ex. convey ideas or arguments in clear, concise, well organized papers Oral communication- ex. convey ideas in cogent, persuasive, and organized presentations; ex. demonstrate comprehensive knowledge of the field, and ability to persuasively present arguments and points of view in discussion Scholarship ex. to produce original, scholarly contributions to field; ex. produce original ideas or arguments that demonstrate in-depth understanding of the theoretical foundations, methods, and scholarship of the field Performance- ex. to develop and present creative works using theory and methods of the field, and present those works in performance Professional ethics- ex. to appreciate and demonstrate a responsible, ethical manner in professional work 4

Teamwork- ex. demonstrate ability to work with others on projects, including sharing work involved in development of initial ideas and project plan, discussion of progress, and completion of work Teaching- ex. prepare and present well organized lectures, classroom activities and assignments that support student learning Leadership- ex. demonstrate leadership through positions held in scholarly and other professional activities Innovation/ entrepreneurship- ex. demonstrate abilities in innovation/ entrepreneurship through development of new inventions, patents, publications, productions, performances, public or private organizations Global citizenship- ex. demonstrate appreciation for one s role as a member of an increasingly connected global society Additional examples with elaborated program learning outcome statements are included in Appendix 9. Examples for close reading, research, critical thinking, writing, and professional development are included. 4. Choose assessment methods that best measure student learning outcome Assessment methods, such as exams, thesis review, juried presentations, or student surveys permit the gathering of direct and indirect evidence with which to measure each program learning outcome. There are two kinds of assessment measures, direct and indirect. Measures can be implemented inside and external to the program. Measures also take place internal and external to the program: Direct measures- permit the gathering of objective data by expert reviewers, eg. qualifying exams, thesis review by faculty. Indirect measures- gather information from the perspective of constituents, eg. alumni or enrolled graduate student surveys. While not objective, the information gathered provides useful insight on program learning processes. Internal measures- controlled by the program, such as qualifying exams. External measures- controlled by groups outside of the institution, such as peer reviewed publications. One can then define measures as direct/ internal and direct/ external, indirect/ internal, and indirect/ external. It is useful to implement, at minimum, three measures for each program learning outcome. Of these, at least 1 should be a direct method while 2 can be indirect methods. Examples of direct and indirect methods are: 5

Direct method, internal to program Course final exams, projects, or presentations Qualifying exam- written Qualifying exam- oral Review of thesis- student s written document Review of thesis- student s oral presentation Review of performance or other creative works Scholarly awards, fellowships and distinctions from internal sources For master s degree students, one can select major final exams or major papers in core, required courses as direct, internal measures, as long as those exams are directly related to particular program learning outcomes. Direct method, external to program Peer reviewed conference and journal publications Review of performance or other creative works Scholarly awards, fellowships and distinctions from external sources Internship evaluation Indirect method, internal to program Graduate student survey of learning while enrolled or at point of graduation (note: plan in place for new UR graduate student survey) Graduate student focus group using targeted questions about learning and professionalization process Indirect method, external to program Alumni survey of learning (note: plan in place for new UR alumni survey) Review of program by advisory board made up of academic and industry members in field Career data- industry, academic positions taken by graduates Advisory board of academic, industry and alumni involved in program review Leadership positions taken by alumni 5. Design each chosen direct and indirect assessment method to ensure measurement quality, and set criterion / benchmark for success The assessment processes of graduate degree programs, such as qualifying exams or dissertation defense, are strong, direct assessment methods. This section provides suggestions for elaborating on such assessment methods so that program learning outcomes are clearly measured by each method. Doctoral degree programs, in general, include the following direct assessment methods: Qualifying exams, written and oral, in which students are tested on their comprehensive knowledge of the core (and specialization) theories, methods, and scholarship in the field 6

and skills in using this body of knowledge. The test is written and student responses reviewed by faculty who oversee each core and specialization area. Defense of dissertation proposal, written and oral, in which students are tested on their abilities to develop and defend a proposal for an original scholarly research work before a review committee. Defense of dissertation, written and oral, in which students are tested on their abilities to defend a completed an original scholarly research work before a review committee Master s degree programs, in general, include the following assessment methods: Master s thesis proposal, written, which is reviewed by faculty thesis advisor or committee Master s thesis, written and oral, which is reviewed by faculty thesis advisor or committee Co-op/ practice project report which is reviewed by internship supervisor or faculty advisor While assessment methods typically used by doctoral and master s degree program are strong, it may not be apparent to students or external reviewers which program learning outcomes are assessed by each method. Therefore, it is suggested that programs develop rubrics (a variation on program learning outcomes in their degree of detail) with which to score student work for a given assessment method, such as a dissertation defense. An example of a dissertation proposal review rubric is shown below. One will notice that it encompasses a number of program learning outcomes of a typical doctoral degree program. It is also sufficiently general to be of use in many disciplines: 1. Problem Definition: States the research problem clearly, providing motivation for undertaking the research 2. Literature and Previous Work: Demonstrates sound knowledge of literature in the area, and of prior work on the specific research problem 3. Impact of Proposed Research: Demonstrates the potential value of solution to the research problem in advancing knowledge within the area of study 4. Solution Plan: Provides a sound plan for applying state-of- the-field research methods/tools to solving the defined problem and shows a good understanding of how to use methods/tools effectively 5. Expected Results: Provides a sound plan for analyzing and interpreting research results/data 6. Quality of Written and Oral Communication: (a) Communicates research proposal clearly and professionally in both (a) written and (b) oral form 7. Critical Thinking: Demonstrates capability for independent research in the area of study, preparedness in core disciplines relevant to research, and ability to complete the proposed research 8. Broader Impact: Demonstrates awareness of broader implications of the proposed research. Broader implications may include social, economic, technical, ethical, business, etc. aspects. Using the above rubric, one would create a scoring system that defines poor, adequate, and superior student performance for each factor. The scale can be as simple as a 3 point scale, or 7

even a 7 or 10 point scale that provides a more nuanced definition of poor to superior student performance. In practice, one might create a Dissertation Proposal Review Form that a student s dissertation committee would use in scoring student performance. Appendices 4-6 present examples of master s thesis and dissertation proposal, dissertation defense review forms that are simple to use. It is suggested that these forms can be adapted for use by UR graduate degree programs and include other important program learning outcomes. Completed forms can be stored in each student s file for later program review. Rubrics with scoring systems can be elaborate. Appendix 7 includes an example of such: a thesis scoring rubric developed by Princeton s Biology Department. It includes rubrics for lab and nonlab based theses. It is included in order to show how faculty might want to try out writing rubrics which are a better fit for their degree programs. Another simple rubric, developed for science and engineering programs by University of Virginia, is included in Appendix 8. 6. Set criterion/ benchmark for success for each assessment method Once assessment methods are chosen, it is important to set the criterion for success for each measure. For example, the program might set the benchmark for each section of the qualifying exam: 100% of students will pass each section of the exam. If all students do not pass a given section, the program might make changes in core courses that prepare students for the exam. 7. Determine an implementation plan and timeline for each assessment method Assessment methods should be embedded into typical program assessment operations. For example, faculty can complete a score sheet which rates student performance after review of a student thesis or presentation. The completed forms could be placed in the student s file for later program review. The College s Director of Assessment will be developing new surveys, UR Graduate Student Survey and UR Alumni Surveys. Department input will be gathered in development of the surveys. The surveys will include important program quality and learning questions that will be useful in program assessment planning. Annual reports will be distributed to the program. NOT ALL PROGRAM LEARNING OUTCOMES NEED TO BE MEASURED USING EACH METHOD OR REVIEWED EVERY YEAR. It is important to create a reasonable timeline for method implementation review; for example, a program use the dissertation review forms every other year. Or the program can review data for program learning outcomes 1 and 2 one year, and program learning outcomes 3 and 4 the following year. 8. Determine plan for reporting and review of assessment data, and how best to connect review with curricular change Once the assessment data is collected, it must be reviewed. An assessment plan report should note data for each program learning outcome in an easy to use form. A committee or group of faculty should be given the responsibility for annual data review. 8

It is essential to organize reports of program assessment data so that faculty can easily review all data for each program learning outcome in a given review. A program assessment report for a given program learning outcome might take the following form. One would then complete additional reports for other program learning outcomes: Program learning outcome 1: Students will demonstrate comprehensive, in depth knowledge of the theories, methods, and scholarship in the field Program assessment method 1: student scores on qualifying exam organized by exam sections Program assessment method 1 results: In AY2009-2010, average student score for historical methods was 100% pass; in AY2010-2011, average student score for historical methods was 70% pass Program standard for success for assessment method 1 and review: 100% of students will pass. Standard not met in AY2010-2011. Program assessment method 2: student publications in peer reviewed journals Program assessment method 2 results: in AY2009-2010, 3 out of 10 students published in peer reviewed journals; in AY2010-2011, 7 out of 10 students published. Program standard for success for assessment method 1 and review: all students will publish in peer reviewed journal or conference. Standard not met in AY2009-2010 or AY2010-2011. 9. Pull it all together in a complete program assessment plan using a plan template Once steps 1 8 above are completed, it is useful to pull together the information in an organized manner for presentation to students, advisory board members, faculty, and accreditation agencies. A commonly used format is the assessment plan template presented in Appendices 1 3. Appendices 1 and 2 present complete sample assessment plans for a typical doctoral degree program and master s degree program, respectively. Appendix 3 presents a blank assessment plan template for program use. It is suggested that programs print out the samples in Appendices 1 and 2 and use as guidelines in preparing their own program template. The Director of Assessment, Barbara Masi, barbara.masi@rochester.edu, is available to support programs in designing an assessment plan, and completing the template. 9

Appendices 1. University of Rochester Doctoral Degree Program Assessment Plan Sample 2. University of Rochester Master s Degree Program Assessment Plan Sample 3. University of Rochester Doctoral or Master s Degree Program Assessment Plan Template 4. Sample dissertation proposal review rubric (all majors) 5. Sample dissertation defense review rubric (all majors) 6. Sample Master s thesis review rubric (all majors) 7. Princeton Univ. Biology PhD program dissertation review rubric 8. Univ. Virginia engineering and science dissertation defense review rubric 9. Examples of expanded program learning outcome statements for professional development, effective researcher, critical thinking, close reading of texts, writing 10