University of Toronto Quality Assurance Process (UTQAP)

Transcription

1 University of Toronto Quality Assurance Process (UTQAP) Revised version approved by the Ontario Universities Council on Quality Assurance September 21, 2012

2 Table of Contents 1 Quality Assurance Context 1.1 Overview 1.2 Institutional authority 2 New Degree Program Approval Protocol 2.1 Purpose and application 2.2 Overview of the program approval process 2.3 Evaluation criteria 2.4 Initial institutional process 2.5 Initial Quality Council appraisal process 2.6 Quality Council appraisal process continuation 2.7 Subsequent process 2.8 Quality Council audit process 3 Major Modifications to Existing Programs Protocol 3.1 Definition 3.2 Proposal 3.3 Institutional process and approvals 3.4 Annual report to the Quality Council 3.5 Subsequent University process 4 Program Closure 4.1 Proposal 4.2 Institutional process and approval 4.3 Annual report to the Quality Council 5 Cyclical Program Review Protocol 5.1 Purpose and application 5.2 Institutional authority 5.3 Degree programs and review schedule 5.4 Commissioning officer 5.5 Overview of the review process 5.6 Self-study requirements: Internal program perspective 5.7 External evaluation: Reviewer selection and review process 5.8 Institutional perspective and response 5.9 Quality Council audit process University of Toronto UTQAP 2

3 1 Quality Assurance Context 1.1 Overview The University of Toronto is committed to being an internationally significant research university, with undergraduate, graduate, and professional programs of excellent quality. i Hence, the University welcomes the opportunity provided by the Ontario Council of Academic Vice-Presidents Quality Assurance Framework (QAF) ii assigning the responsibility for academic standards, quality assurance and program improvement, in the first instance, to universities themselves. The University of Toronto s approach to quality assurance is built on two primary indicators of academic excellence: (1) the quality of the scholarship and research of faculty and (2) the success with which that scholarship and research is brought to bear on the achievement of Degree Level Expectations. These indicators are assessed by determining how our scholarship, research and programs compare to those of our international peer institutions and how well our programs meet their Degree Level Expectations. Reviews provide the opportunity to celebrate successes, identify areas where we can do better, and vigorously pursue improvements. The Policy for Approval and Review of Academic Programs and Units governs the approval of proposed new programs and the review of existing programs at the University of Toronto. The University of Toronto Quality Assurance Process (UTQAP) outlines the protocols for the assessment and approval of new programs, review of existing programs, modifications to existing programs, and closures of programs. Complementing this document, the University has developed a series of standardized templates to support the quality assurance process. These and a wide range of explanatory materials and best practice exemplars are available on the Vice-Provost, Academic Programs UTQAP website. The Policy for Approval and Review of Academic Programs and Units was approved by the Governing Council of the University of Toronto on June 24, The UTQAP was brought forward for information at that time and was subsequently ratified by the Ontario Universities Council on Quality Assurance (the Quality Council) on March 31, The current version of the UTQAP contains a number of small revisions to ensure greater clarity and to bring the document in line with evolving practice across the province following the first full year under the Quality Assurance Framework. It was approved by the Quality Council on September 21, The University of Toronto s responsibilities for quality assurance extend to new and continuing undergraduate and graduate degree and diploma programs whether offered in full or in part by the UofT, or conjointly with any institutions federated or affiliated with the University. These responsibilities also extend to programs offered in partnership, collaboration or other such arrangement with other postsecondary institutions including colleges, universities, and institutes. The Quality Council ensures that Ontario continues to maintain a rigorous quality assurance framework. It ratifies each institution s Quality Assurance Process [IQAP] and is responsible for approving any subsequent revisions to that IQAP. It also is responsible for conducting an audit of university processes through a panel of auditors that reports to a committee of the Council. The panel s role is to examine each institution s compliance with its own Quality Assurance Process. The Quality Council approves and monitors the audit reports. The University of Toronto Quality Assurance Process (UTQAP) encompasses four elements: The New Degree Program Approval Protocol applies to new undergraduate degrees, undergraduate specialists and majors, graduate programs and degrees, graduate diplomas, and collaborative graduate programs. The Quality Council has provided the following statement regarding the definition of new programs: To clarify, for the purposes of the Framework, a 'new program' is brand-new: that is to say, the program has substantially different program requirements and substantially different learning outcomes from those of any existing approved programs offered by the institution. University of Toronto UTQAP 3

4 New programs and degrees are externally reviewed as part of the process leading to approval by institutional governance. Proposals for graduate diplomas and collaborative programs do not require external appraisal. Once approved by University governance, these new program proposals are assessed by the Appraisal Committee of the Quality Council. This Council has the authority to approve or decline all new program proposals. The Major Modification Protocol is used to ensure program quality where major substantive changes are made to existing and previously approved programs. Major modifications are approved through University governance processes and are reported annually to the Quality Council. The Program Closure Protocol articulates a process for closing programs. There are a number of possible reasons for closing a program including low enrolment, changes in the disciplinary landscape, and poor quality of the academic program. These reasons may be articulated in external review reports or may be identified by members of the University community. Program closures are approved through University governance processes and are reported annually to the Quality Council. The Cyclical Program Review Protocol ensures the quality of existing undergraduate and graduate degree programs and for-credit graduate diplomas. The review of an academic program may be a part of a review of the academic unit(s) in which the program resides. In addition to the protocols described in the UTQAP, the Vice-Provost, Academic Programs UTQAP website: a) provides templates that establish formats for new program proposals, major modifications, program closures, self-studies and external review reports; b) describes best practices and establishes criteria for administrative processes such as the selection of reviewers and scheduling of appraisals of new and existing programs and units; c) provides guidance on the conduct of self-studies; d) identifies responsibilities for the collection, aggregation and distribution of standardized data and outcome measures required for self-studies; e) sets out the University s cycle for the conduct of undergraduate and graduate program reviews; and f) establishes contact information for support and assistance. 1.2 Institutional authority The Vice-President and Provost is the chief academic officer and chief budget officer at the University of Toronto. The Provost, with the Vice-Provost, Academic Programs, is responsible for the oversight of the University of Toronto Quality Assurance Process and ensuring that the UTQAP is applied in a manner that conforms to the UofT s quality assurance principles and to Quality Council requirements. Within the Office of the Vice-Provost, Academic Programs, the Director, Academic Programs and Policy is the contact between the institution and the Quality Council. New Degree Program Proposals: The Office of the Vice-Provost, Academic Programs responds to divisional queries and facilitates proposal development with respect to University of Toronto UTQAP 4

5 institutional academic, planning and budget, student life, governance and approval aspects of proposals. Major Modifications to Existing Programs: The Office of the Vice-Provost, Academic Programs consults with divisions on the development of proposals for major modifications to existing programs. The Office receives copies of approved program modifications and compiles an annual report of all divisional modifications. Program Closures: The Office of the Vice-Provost, Academic Programs responds to divisional queries and facilitates the development of proposals for the closure of programs. The Office includes program closures in the annual report to the Quality Council. Cyclical Reviews: The Office of the Vice-Provost, Academic Programs is responsible for ensuring that cyclical reviews of academic programs and/or units are undertaken. Where quality concerns are raised in the cyclical review, the Vice-Provost, Academic Programs monitors the timely implementation of improvements. The Office of the Vice-Provost, Academic Programs maintains a UTQAP website that includes information pertaining to the Quality Assurance process, all related templates and materials, program approval and review schedules, and contact information. University of Toronto UTQAP 5

6 2 New Degree Program Approval Protocol The primary responsibility for the design and quality assurance of new undergraduate and graduate degree programs lies with the University and its governing bodies. Academic divisions are responsible for curriculum design, the identification of program objectives, the development of learning outcomes and degree level expectations, and the assembly of human, instructional and physical resources. The approval protocol helps to ensure that programs are aligned with the objectives of the academic division and of the University as specified within the Statement of Institutional Purpose and thereby advance the mission of the University and the academic division. 2.1 Purpose and application The New Degree Program Approval Protocol sets out the steps to be taken at the University to assemble and provide the information required in support of new program proposals. The purpose of the Protocol is to ensure that the procedures followed for the assessment of proposed new academic degree programs is in accordance with the University Policy for Approval and Review of Academic Programs and Units and the provincial Quality Assurance Framework. The New Degree Program Approval Protocol applies to the development of new undergraduate or graduate degrees, undergraduate specialists and majors within approved degrees and to graduate degree programs, graduate collaborative programs, and diplomas, offered in full or in part by the UofT or by the UofT jointly or conjointly with institutions federated or affiliated with the University: New Degree Program Proposals are assessed within the division and by the Office of the Provost as part of the program development process prior to external appraisal and submission to University governance. The program proposal must address the purpose and content of the new program and the capacity of the unit to deliver a high quality program. The Dean is responsible for commissioning the external appraisal of proposed new programs with the approval of the Vice-Provost, Academic Programs. Programs that are inter- and multidisciplinary must identify a permanent lead administrative division and identify a commissioning officer for future cyclical program reviews. Programs that are inter-institutional and offered jointly, conjointly and/or in affiliation with other higher education institutions (colleges and universities) through formal agreements are assessed as entities distinct from the larger institutions within which they are included. Where a program is held jointly with an Ontario institution that does not have an IQAP that has been ratified by the Quality Council, the UTQAP will serve as the guiding document and University of Toronto will be the lead institution. Where a program is held jointly with an Ontario institution that does have an IQAP that has been ratified by the Quality Council, a lead institution will be selected. Program proposals specify how future reviews will be conducted. 2.2 Overview of the program approval process The steps required to develop and approve proposals for new undergraduate degrees, undergraduate specialists or majors within existing degrees, graduate programs and degrees, graduate diplomas, and graduate collaborative programs, are indicated in Figures 1a (standard approval) and 1b (expedited approval). New undergraduate degrees, undergraduate specialists or majors, graduate degrees and programs are subject to the full standard approval process which includes an external appraisal. New graduate diplomas and collaborative graduate programs may be brought forward under an expedited process which requires the submission of a proposal to the Quality Council but does not require an external appraisal. University of Toronto UTQAP 6

7 Figure 1a: Standard process for approval of new undergraduate and graduate degrees and programs 1. INTERNAL UNIVERSITY PROCESS Division: Proposal initiation Provost s Office (2.4.2): All programs (at outline stage) are brought to the Vice-Provost, Academic Programs who responds to divisional queries and facilitates proposal development through consultation with other Vice-Provostial portfolios. Vice-Provost, Academic Programs and/or Vice-Provost, Graduate Education considers: Program rationale including consistency with the unit s academic plan Appropriateness of the name and degree designation Program description, requirements, content and standards; program objectives; learning outcomes; faculty and teaching staff requirements and supervisory capacity Impact on the nature and quality of the division's programs of study Impact on other divisions and need for inter-divisional and inter-institutional consultation and agreements/contracts Vice-President, University Operations considers: Resource implications, including, but not limited to, staffing, libraries and computing facilities, enrolment/admissions, revenue/costs, financial aid Enrolment planning, revenue and expense projections BIU eligibility Space allocations and operating costs; capital project approvals Vice-President, University Relations MTCU program approvals process and submission requirements Vice-Provost, Students and/or Vice-Provost, Graduate Education considers: Impact on student affairs and services; registrarial and information systems; awards and admissions Vice-Provost, Faculty and Academic Life considers: Faculty implications Division: Proposal development Broad consultation: with faculty, students, other academic divisions, and external stakeholders ` Dean s Office and Provost s Office signoff (2.4.3) Division: External appraisal commissioned (2.4.4) Division and Provost s Office: Internal response to appraisal (2.4.6) Divisional Governance Approval (2.4.7) Provost s Office: Submits proposal for University Governance Approval (2.4.7) New specialists and majors are approved at the level of AP&P. New undergraduate degrees, graduate degrees and programs are recommended by AP&P to Academic Board and confirmed by the Executive Committee of Governing Council 2. QUALITY COUNCIL APPROVAL PROCESS Provost s Office: Submits proposal to the Quality Council (2.4.8) Appraisal Committee Review and Recommendation(2.5.2) (normally within 45 days of receipt of the institution s submission) Quality Council Approval to commence 3. MTCU PROCESS University: Submission to MTCU if new degree or new graduate degree or program (2.7.1) 4. FOLLOW- UP PROCESS Ongoing program monitoring by the University (2.7.3) Cyclical Review within 8 years of first enrolment University of Toronto UTQAP 7

8 Figure 1b: Expedited process for approval of new graduate diplomas and graduate collaborative programs 1. INTERNAL UNIVERSITY PROCESS Division: Proposal initiation Provost s Office (2.4.2): All programs (at the outline stage) are brought to the Vice-Provost, Academic Programs who responds to divisional queries and facilitates proposal development through consultation with other Vice-Provostial portfolios. Vice-Provost, Academic Programs and/or Vice-Provost, Graduate Education considers: Program rationale including its consistency with the unit s academic plan. Appropriateness of the name and degree designation. Program description, requirements, content and standards; program objectives; learning outcomes; faculty and teaching staff requirements and supervisory capacity. Impact on the nature and quality of the division's programs of study. Impact on other divisions and need for inter-divisional and inter-institutional consultation and agreements/contracts. Vice-President, University Operations considers: Resource implications, including, but not limited to, such areas as staffing, space, libraries and computing facilities, enrolment/admissions, revenue/costs, financial aid. Enrolment planning; revenue and expense projections. BIU eligibility. Space allocations and operating costs; capital project approvals. Vice-President, University Relations MTCU program approvals process and submission requirements Vice-Provost, Students and/or Vice-Provost, Graduate Education considers: Impact on student affairs and services; registrarial and information systems; awards and admissions. Vice-Provost, Faculty and Academic Life considers: Faculty implications. Division: Proposal development Broad consultation: with faculty, students, other academic divisions, and external stakeholders Dean s Office and Provost s Office signoff (2.4.3) Divisional Governance Approval (2.4.7) Provost s Office: Submits proposal to University Governance Approval: AP&P (2.4.7) 2. QUALITY COUNCIL APPROVAL PROCESS Provost s Office: Submits proposal to the Quality Council (2.4.8) Expedited Approval Process: Appraisal Committee Review and Recommendation (normally within 45 days of receipt of the institution s submission) Quality Council Approval to commence 3. MTCU PROCESS University: Submission to MTCU if new diploma 4. FOLLOW- UP PROCESS Ongoing program monitoring by the University Cyclical Review within 8 years of first enrolment University of Toronto UTQAP 8

9 2.3 Evaluation criteria identified in the Quality Assurance Framework Proposals for new graduate or undergraduate degree programs are evaluated against the following criteria set by the Quality Assurance Framework. Academic divisions are responsible for the development of a New Program Proposal that addresses the evaluation criteria below together with any further divisional requirements which the academic division chooses to apply (see UTQAP New Program Templates) Objectives a) Consistency of the program with the institution s mission and unit s academic plans. b) Clarity and appropriateness of the program s requirements and associated learning outcomes in addressing the academic division s undergraduate or graduate Degree Level Expectations. c) Appropriateness of degree or diploma nomenclature Admission requirements a) Appropriateness of the program s admission requirements for the learning outcomes established for completion of the program. b) Sufficient explanation of additional requirements, if any, for admission into a graduate, second-entry or undergraduate program, such as minimum grade point average or additional languages or portfolios, along with how the program recognizes prior work or learning experience Structure a) Appropriateness of the program's structure and regulations to meet specified program learning outcomes and Degree Level Expectations. b) For graduate programs, a clear rationale for program length that ensures that the program requirements can be reasonably completed within the proposed time period Program content a) Ways in which the curriculum addresses the current state of the discipline or area of study. b) Identification of any unique curriculum or program innovations or creative components. c) For research-focused graduate programs, clear indication of the nature and suitability of the major research requirements for degree completion. d) Evidence that each graduate student in the program is required to take all of the course requirements from among graduate level courses. iii Mode of delivery a) Appropriateness of the proposed mode(s) of delivery (distance learning, compressed part-time, online, mixed-mode or non-standard forms of delivery, flex-time options) to meet the intended program learning outcomes and Degree Level Expectations. University of Toronto UTQAP 9

10 2.3.6 Assessment of teaching and learning a) Appropriateness of the proposed methods for the assessment of student achievement of the intended program learning outcomes and Degree Level Expectations. b) Completeness of plans for documenting and demonstrating the level of performance of students, consistent with the academic division s statement of its Degree Level Expectations Resources for all programs a) Adequacy of the administrative unit s planned utilization of existing human, physical and financial resources, and any institutional commitment to supplement those resources to support the program. b) Participation of a sufficient number and quality of faculty who are competent to teach and/or supervise in the program. c) Evidence that there are adequate resources to sustain the quality of scholarship and research activities of undergraduate and graduate students, including library support, information technology support, and laboratory access. d) A budget outline including proposed enrolment, proposed tuition, and indication of whether the proposed program will be cost-recovery Resources for graduate programs only a) Evidence that faculty have the recent research or professional/clinical expertise needed to sustain the program, promote innovation and foster an appropriate intellectual climate. b) Where appropriate to the program, evidence that financial assistance for students will be sufficient to ensure adequate quality and numbers of students. c) Evidence of how supervisory loads will be distributed, and the qualifications and appointment status of supervisors Resources for undergraduate programs only a) Evidence of and planning for adequate numbers and quality of faculty and staff to achieve the goals of the program. b) Planning and commitment to provide the necessary resources in step with the implementation of the program. c) Planned/anticipated class sizes. d) Provision of supervision of experiential learning opportunities (if required). e) The role of adjunct and part-time faculty. University of Toronto UTQAP 10

11 Quality and other indicators a) Definition and use of indicators that provide evidence of the quality of the faculty (e.g., qualifications, research, innovation and scholarly record; appropriateness of collective faculty expertise to contribute substantively to the proposed program). b) Evidence of a program structure and faculty research that will ensure the intellectual quality of the student experience. 2.4 Initial institutional process Institutional authority and Quality Council contact The Provost with the Vice-Provost, Academic Programs is responsible for the oversight of the University of Toronto Quality Assurance Process and ensuring that the UTQAP is applied in a manner that conforms to the University s quality assurance principles and Quality Council requirements. The Office of the Vice-Provost, Academic Programs responds to divisional queries and facilitates proposal development with regard to institutional academic, planning and budget, student life, governance and approval aspects of proposals. Within the Office of the Vice-Provost, Academic Programs, the Director, Academic Programs and Policy is the contact between the institution and the Quality Council New Program Proposal development and submission to the Vice-Provost, Academic Programs New programs are initiated within academic divisions. The Office of the Dean of the academic division submits the initial proposal outline to the Vice-Provost, Academic Programs who is responsible for providing feedback regarding the program including input from the Provost and other Vice-Provosts, as appropriate. Once the program has been approved for development, the division works with the Office of the Provost to develop the New Program Proposal. The Dean ensures appropriate compliance with the evaluation criteria (Section 2.3) and ensures that appropriate consultation is conducted with the Office of the Vice-Provost, Academic Programs early in the process of proposal development. The Dean ensures that appropriate consultation is conducted with faculty and students, other university divisions and external institutions. The Dean commissions the external appraisal of a new program as required with the approval of the Vice-Provost, Academic Programs. The Office of the Provost reviews and approves draft proposals as identified in Figures 1a and 1b Program proposal The Vice-Provost, Academic Programs confirms that the New Program Proposal is complete and includes information on all the evaluation criteria (Section 2.4.2), so that the submission process can continue External appraisal iv An external appraisal is required for new undergraduate and graduate degrees, new undergraduate specialists and majors, and new graduate degree program proposals only. The following process is required in the selection and appointment of external reviewers who review a New Program Proposal. University of Toronto UTQAP 11

12 The external appraisal of a New Program Proposal is commissioned by the Dean of the relevant academic division with the approval of the Vice-Provost, Academic Programs. There must be at least one reviewer for a new undergraduate program and two for a new graduate program. The reviewers should be active and respected in their disciplines, and will normally be associate or full professors, or the equivalent, with program management or senior academic administrative experience. They must be at arm s length from the program under appraisal. (See the UTQAP website for a definition of arm s length, suggestions on the selection of reviewers, and a reviewer nomination form) The external appraisal of a new graduate program proposal must incorporate an onsite visit. The external appraisal of a new undergraduate program proposal is normally conducted on-site, but may be conducted by desk audit, video-conference or an equivalent method if the external reviewer is satisfied that the off-site option is acceptable. (The UTQAP website includes sample instructions to reviewers.) The external reviewers provide a joint report that appraises the standards and quality of the proposed program Appraisal Report The reviewers provide a joint report evaluating the standards and quality of the proposed program and make recommendations for any essential or desirable modifications to it. This is normally presented within two weeks of the site visit. As part of the process, reviewers are invited to acknowledge any clearly innovative aspects of the proposed program Administrative Responses An Administrative Response to the New Program Proposal and Appraisal Report is required from the Dean of the proposing academic division following consultation with the academic unit proposing the program University of Toronto approval The New Program Proposal, the External Appraisal Report and the internal Administrative Responses proceed through the divisional and university governance processes. Divisional governance Each academic division is responsible for delineating governance approval processes for new undergraduate and graduate programs/diplomas. The Vice-Provost, Academic Programs is responsible for reviewing these processes and ensuring compliance with University and UTQAP processes. Each division outlines its process on its own council website. A summary of divisional governance processes is available on the website of the Vice-Provost, Academic Programs. University-wide governance Proposals are submitted to university governance through the Provost's Office, which recommends items to the Committee on Academic Policy and Programs and Academic Board through their Senior Assessors. Upon approval of a new program by divisional council, the New Program Proposal, Appraisal Report, and Administrative Responses are submitted to the Committee on Academic Policy and Programs by the Vice-Provost, Academic Programs. The Committee on Academic Policy and Programs approves proposals for new undergraduate programs and recommends proposals for new undergraduate degrees and graduate programs to Academic Board for final approval. University of Toronto UTQAP 12

13 2.4.8 Quality Council Secretariat Upon approval by University governance, the Office of the Vice-Provost, Academic Programs submits the New Program Proposal, together with all required reports and documents, to the Quality Council Announcement of new programs Following the submission of the New Program Proposal to the Quality Council, the academic unit may announce its intention to offer the program, provided that clear indication is given that approval by the Quality Council is pending and provided that no offers of admission will be made until and unless the program is approved by the Council. 2.5 Initial Quality Council appraisal process Secretariat check The Quality Council Secretariat will confirm that the New Program Proposal and associated reports and internal responses to them (as set out in Section 2.4 above) are complete. If there is missing information or defects of substance, the Secretariat will return the New Program Proposal for revision or amendment and resubmission. Otherwise the proposal and accompanying documents will be forwarded directly to the Quality Council Appraisal Committee Appraisal Committee reviews and recommends The Quality Council s Appraisal Committee reviews and appraises the complete file. This committee may seek further information, in which case it provides reasons for its requests. In rare instances, the Appraisal Committee may invite further input from an external expert, either through a desk audit or site visit. If no further information is required, the Appraisal Committee, through the Quality Council, will propose its recommendation, including a brief explanation of its reasons. This assessment includes one of the following recommendations: a) Approval to commence; b) Approval to commence, with report; (This typically refers to some provision or facility not currently in place but planned for later implementation, often two to three years in the future. The with report condition implies no lack of quality in the program, does not hold up the implementation of the new program, and is not subject to public reference, whether on the web or elsewhere.) c) Deferral for up to one year during which time the university may address identified issues and report back; or d) Against approval. This step will normally be completed within forty-five days of receipt of the University s submission, provided that the submission is complete and in good order, and that no further information or external expert advice is required. Where additional information is required by the Appraisal Committee, one of the four possible recommendations (see above) to the Council will be made within a further thirty days of its receipt. University of Toronto UTQAP 13

14 2.6 Quality Council appraisal process continuation Institution may consult/appeal to Committee When the recommendation is one of b), c) or d) in above, the University may, within sixty days, make an appeal to, or request a meeting with, the Appraisal Committee for reconsideration. Normally, the grounds for seeking reconsideration are that the University will be providing new information; that there were errors of fact in the Appraisal Committee s commentary; or that there were errors of process. Following such communication, the Appraisal Committee revisits and may revise its assessment. It will convey its final recommendation to the Quality Council Institution may appeal to Council. Council decides Having received and considered the Appraisal Committee s final assessment and recommendation and any additional comments from the University on the assessment, and having heard any requested appeal from the University on matters of fact or procedure, the Council makes one of the following decisions: a) Approved to commence; b) Approved to commence, with report; c) Deferred for up to one year, affording the University an opportunity to amend and resubmit its Proposal; or d) That the Program Proposal is declined. When the Quality Council chooses option c), then the Appraisal Committee suspends the assessment process until the University has resubmitted its Proposal. After this, the Appraisal Committee reactivates its appraisal process (see Section above). When the Appraisal Committee does not receive a response within the specified period, it considers the Proposal to have been withdrawn Council reports decision The Quality Council conveys its decision to the University through the designated institutional contact, and reports it for information to the Ontario Council of Academic Vice- Presidents (OCAV) and to the Ministry of Training, Colleges and Universities (MTCU). Information about decisions on approval to commence for new programs, together with a brief description of the programs, are posted on the websites of the Quality Council and the Vice-Provost, Academic Programs. Only at this point may the University make offers of admission to the program Waiting period before resubmission To allow time for revisions to proposals, any institution declined permission to proceed at this stage of the process, or following a denied appeal of the decision, will normally wait until one year has elapsed from the date of the Quality Council s decision before resubmitting a revised version of its proposal. The same waiting period normally applies when a university does not resubmit a deferred program proposal within the specified period. University of Toronto UTQAP 14

15 2.6.5 Subsequent appraisal with report When the University has been given approval to commence a program with report, the Appraisal Committee reviews the subsequently submitted report, conducts whatever consultation it requires, and then makes one of the following recommendations to the Council. That: a) The program be approved to continue without condition. b) The program may continue accepting admissions, but the Council requires additional follow-up and a report within a specified period, prior to the conduct of the initial cyclical review. On the Council s receipt of that required report, the procedure returns to this same step in the appraisal process (i.e., Section 2.6.6). c) The program be required to suspend admissions for a minimum of two years. The Quality Council will then specify the conditions to be met in the interim in order for admissions to the program to resume. d) The University may appeal, to the Quality Council, the proposed recommendation of the Appraisal Committee to suspend admissions to the program (Section 2.6.5c) on the same terms as are set out in Section above (i.e., the University will be providing new information; and/or there were errors of fact in the Appraisal Committee s commentary; and/or there were errors of process) Council hears appeal based on report. Council decides Having received and considered the Appraisal Committee s recommendation, and the University s appeal, if any, the Quality Council may decide: a) To approve the program without condition, or b) To approve the program continuing admissions with a further report, or c) To require the program to suspend admissions for a minimum of two years. This decision is final. The Quality Council conveys its decision to the University, and reports it to OCAV and to MTCU for information. 2.7 Subsequent process Ministry of Training, Colleges and Universities (MTCU) funding approval for new undergraduate degrees and graduate degrees and programs The Minister approves funding (BIUs) for new degree and diploma programs. The approval process occurs several times per year. Proposals are submitted to MTCU by the University once Quality Council approval has been received Implementation window After a new program has been approved to commence, the program must begin within 36 months of that date of approval; otherwise the approval will lapse On-going monitoring of new programs It is the responsibility of the Dean, in consultation with the head of the relevant academic units, to monitor student enrolment and success in the program, as well as resource allocation and program administration. As part of the annual academic review process, the Office of the Vice-President and Provost works with Deans Offices to review the quality and performance of all program offerings and address any areas of concern. University of Toronto UTQAP 15

16 2.7.4 First cyclical review The first cyclical review for any new program must be conducted no more than 8 years after the date of the program s initial enrolment and normally in accordance with the UofT program review schedule. The Dean is responsible for conveying to the Vice-Provost, Academic Programs the inclusion of the program in the University s review schedule. 2.8 Quality Council audit process At least one of the undergraduate programs and one of the graduate programs selected for the sample for each institutional audit (See Quality Assurance Framework Section 5.2.2) will be a New Program or a Major Modification to an Existing Program approved within the period since the conduct of the previous audit. The audit cannot reverse the approval of a program to commence. University of Toronto UTQAP 16

17 3 Major Modifications to Existing Programs Protocol 3.1 Definition A major modification to an existing program is a restructuring of a program, a merger of existing programs or a renewal of a program in order to keep it current with its academic discipline. At the University of Toronto major modifications include one or more of the following program changes: A) Significant changes to program requirements: Creation of a new program of specialization where another with the same designation already exists (e.g., a new specialist program where a major with the same designation already exists) Addition of a new major or specialist that does not differ substantially in program requirements or learning outcomes from an existing program Merger of two or more existing programs Creation of a minor where there is no existing program of specialization The creation of new bridging options for college diploma graduates The introduction or deletion of a thesis requirement, co-op requirement or placement at the undergraduate or graduate level The creation or deletion of a field within an existing graduate program The creation or deletion of a stream within an existing undergraduate program B) Significant changes to the learning outcomes: Changes to program content that affect the learning outcomes, but do not meet the threshold for a new program C) Significant changes to the faculty engaged in delivering the program and/or to the essential physical resources as may occur, for example, where there have been changes to the existing mode(s) of delivery (e.g., different campus, online delivery, inter-institutional collaboration): A change to the language of the program The establishment of an existing degree program at another institution or location Change in mode of delivery of a program such as from classroom to online or fulltime to part-time Major modifications to existing programs do not require submission of a proposal to the Quality Council. The University may request that the Quality Council review a major modification proposal. Normally this will occur through the Expedited Approval Process without the requirement of an external review process. Minor modifications are changes to courses and curriculum that do not change the nature or essence of a program or the learning outcomes. University of Toronto UTQAP 17

18 The University of Toronto considers minor modifications to include: Creation of a new minor within an existing program Changes to admission requirements Creation of a new course Minor changes require approval by divisional governance processes only. In cases where it is unclear whether a proposed change in a program is a new program, a major modification, or a minor modification, a determination will be made by the Vice- Provost, Academic Programs in consultation with the divisional Dean and the academic unit. 3.2 Proposal The proposal for a major modification includes the following together with any additional requirements which the academic division chooses to apply (see the appropriate template on the UTQAP website): Rationale for the major modification and consistency with the unit s academic plan. Outline of the major changes to the program description, requirements, and program learning outcomes. Description of any impact that the major modification may have on students or other divisions; description of consultation with those affected. Description of any resulting resource implications, including, but not limited to, such areas as staffing, space, libraries and computing facilities, enrolment/admissions, and revenue/costs. 3.3 Institutional process and approvals Major modifications to academic programs are initiated within academic divisions. The division s Dean s Office is responsible for the development of a major modification proposal and coordination and consultation with the Office of the Vice-Provost, Academic Programs. The Vice-Provost, Academic Programs is responsible for providing feedback regarding the major modification that includes the input of the Provost and other Vice-Provosts, as appropriate. In particular, major modifications for graduate programs receive special attention from the Vice-Provost Graduate Education. The University of Toronto is responsible for approvals of major modifications to existing programs. Such modifications are normally submitted by the Dean s office for approval by divisional governance. 3.4 Annual report to the Quality Council The Vice-Provost, Academic Programs files an annual report to the Quality Council which provides a summary of major program modifications that were approved through the University s internal approval process in the past year. 3.5 Subsequent University process Cyclical review of the program according to the pre-existing cycle within 8 years. University of Toronto UTQAP 18

19 Figure 2: Process for approval of Major Modifications of undergraduate and graduate programs 1. INTERNAL UNIVERSITY PROCESS Divisional Dean s Office: Proposal development (3.2) Includes consultation with faculty, students, other academic divisions, and external stakeholders as appropriate Consultation with the Vice-Provost, Academic Programs Major modifications of graduate programs will receive special attention from the Vice- Provost, Graduate Education. Dean s Office signoff on major modification Divisional Governance Approval (3.3) Division: Reports approval to the Office of the Vice-Provost, Academic Programs Provost s Office: Submits proposal to Quality Council as part of Annual Report (3.4) 2. FOLLOW-UP PROCESS Ongoing program monitoring by the University through Cyclical Program Review (3.5) University of Toronto UTQAP 19

20 4. Program Closure There are a number of possible reasons for closing a program including low enrolment, a changing disciplinary landscape, and poor quality of the academic program. These reasons may be articulated in external review reports or may be identified by members of the University community. 4.1 Proposal The proposal for a program closure will include the following criteria together with any additional requirements which the academic division chooses to apply (see the Vice-Provost, Academic Programs UTQAP website): Rationale for the closure including alignment with the unit s academic plan. Impact on the nature and quality of the division's program of study. Impact of closure on other units including inter-divisional and inter-institutional agreements/contracts. Impact on and accommodation of any students currently enrolled in the program. 4.2 Institutional process and approvals Proposals for the closure of degrees and degree programs are brought forward along the same governance path as proposals for new programs. Once the Provost s Office has signed off on a proposed closure, the closure is taken forward for approval to the divisional council. Program closures for all components of an undergraduate program are approved by the Committee on Academic Policy and Programs; closures of degrees and all graduate programs are approved by the Academic Board, as recommended by the Committee on Academic Policy and Programs. The closure of one component within an existing undergraduate program or of a minor is considered a major modification and follows the same governance path as proposals for major modifications. 4.3 Annual report to the Quality Council Program closures are reported annually to the Quality Council by the Vice-Provost, Academic Programs. University of Toronto UTQAP 20

21 Figure 3: Process for approvals of program closures 1. INTERNAL UNIVERSITY PROCESS Division: Proposal initiation for program closure (4.1) Provost s Office: All proposals for undergraduate and graduate program closures come to the Provost s Office for preliminary discussion. Graduate programs receive special attention from the Vice-Provost, Graduate Education and Dean of the School of Graduate Studies. These discussions can cover areas such as: Rationale for the closure including alignment with the unit s academic plan. Impact on the nature and quality of the division's program of study. Impact of closure on other units including inter-divisional and inter-institutional agreements/contracts. Impact on and accommodation of any students currently enrolled in the program. Division: Proposal development Broad consultation: with faculty, students, other academic divisions, and external stakeholders Provost s Office signoff for undergraduate and graduate program closures Divisional Governance Approval (4.2) Provost s Office: Submits proposal for University Governance Approval: AP&P and/or Academic Board as appropriate Provost s Office: Reports closure to Quality Council (part of annual report) (4.3) 2. MTCU PROCESS University: Reports closure of degrees to MTCU as part of annual report University of Toronto UTQAP 21

22 5. Cyclical Program Review Protocol 5.1 Purpose and application The Cyclical Program Review Protocol is used to ensure University of Toronto programs meet the highest standards of academic excellence. As stated in the Policy on Approval and Review of Academic Programs, regular reviews allow for ongoing appraisal and quality improvement of programs and the academic units in which they reside. The Cyclical Program Review Protocol applies to all undergraduate and graduate degree programs offered by the University, and to degree programs that are offered by the University with other institutions including all joint, multidisciplinary, interdisciplinary, multisite and inter-institutional programs, and all modes of delivery. 5.2 Institutional authority The Vice-Provost, Academic Programs is responsible for the oversight of the University of Toronto Quality Assurance Process and ensuring that the UTQAP is applied in a manner that conforms to the University s quality assurance principles and Quality Council requirements. The Office of the Vice-Provost, Academic Programs is responsible for ensuring that cyclical reviews of academic programs and/or units are undertaken. Where quality concerns are raised in the cyclical review, the Vice-Provost, Academic Programs monitors the timely implementation of improvements. Within the Office of the Vice-Provost, Academic Programs, the Director, Academic Programs and Policy is the authoritative contact between the institution and the Quality Council. 5.3 Degree programs and review schedule The University s full complement of undergraduate and graduate degree and diploma programs are reviewed on a planned cycle. v Reviews are conducted on a regular basis, frequent enough to ensure that Chairs, Deans, and the Provost are kept informed of developments in all academic units, but at sufficiently long intervals that the effects of given actions can be assessed and that the system is not over-burdened by the logistical demands of the process. The interval between program reviews must not exceed eight years. The review of an academic program can be completed through a review of the academic unit offering the program. Reviews of the various programs, undergraduate and graduate, offered by a given academic unit may be synchronized. Reviews may also be conducted concurrently with professional accreditation. Depending upon the range of a division s academic programs, it can elect to conduct quality reviews at the level of the degree or the program. Regardless of the schedule, the quality of each academic program and the learning environment of the students in each program must be addressed explicitly as set out in the evaluation criteria below. University-commissioned reviews are not waived because an externally-commissioned review, such as an accreditation, has recently been conducted. Reviews of academic programs for professional accreditation bodies form part of collegial self-regulatory systems intended to ensure that mutually agreed-upon threshold standards of quality are maintained in new and existing programs. Such reviews may serve different purposes than those commissioned by the University under the UTQAP. In some cases, however, the University process may be streamlined if the mandates of externally and internally commissioned reviews are closely aligned and any deficits can be easily remedied through providing supplementary documentation as necessary. University of Toronto UTQAP 22

23 Interdivisional degree programs offered by more than one unit may be reviewed as entities distinct from the larger academic units which support them. Such programs must have an identified commissioning division for the purpose of administering the Cyclical Program Review Protocol. Inter-institutional programs offered in partnership with other higher education institutions (colleges and universities) through affiliation, federation and other formal agreements are reviewed as entities distinct from the larger institutions within which they may be included. Where a program is held jointly with an Ontario institution that does not have an IQAP that has been ratified by the Quality Council, the UTQAP will serve as the guiding document and the University of Toronto will be the lead institution. Where a program is held jointly with an Ontario institution that does have an IQAP that has been ratified by the Quality Council, a lead institution will be selected. General guiding principles for such reviews include: Selection of reviewers will involve participation by each partner institution; There will be a single self-study; The site visit will involve all partner institutions and sites; The self-study will clearly explain how input was received from faculty, staff and students at each partner institution; Feedback on the reviewers report will be solicited from participating units at each institution; Preparation of a Final Assessment Report and Implementation Plan will contain input from each partner; A single Final Assessment Report and Implementation Plan will be prepared and presented to the appropriate governance processes at each partner institution; Partner institutions will agree on an appropriate monitoring process for the Implementation Plan. 5.4 Commissioning officer Reviews of academic programs and the units in which they reside are commissioned by the Dean. The Vice-Provost, Academic Programs commissions reviews of academic divisions and associated programs that are being reviewed at the time of a divisional review. A database containing a schedule of all program reviews is maintained by the Office of the Vice-Provost, Academic Programs. See the Vice-Provost, Academic Programs UTQAP website for a schedule of reviews. In the case of programs that involve more than one unit, the review is commissioned by the Dean of the lead faculty. 5.5 Overview of the review process The UTQAP for the conduct of Cyclical Program Reviews has five principal components: a) Self-study (see Section 5.6.4); b) External evaluation (peer review) with report and recommendations on program quality improvement (see Section 5.7); University of Toronto UTQAP 23

24 c) University evaluation of the self-study and the external assessment report resulting in recommendations for program quality improvement (see Section 5.8); d) Preparation and adoption of plans to implement the recommendations and to monitor their implementation (see Section 5.8.3); and e) Follow-up reporting on the principal findings of the review and the implementation of the recommendations (see Section 5.8.4). 5.6 Self-study requirements: Internal program perspective Unit of review The commissioning officer defines the scope of a review (e.g., undergraduate program(s), graduate program(s), etc.) and formally initiates the review process. For example, a unit may elect to review both its undergraduate and graduate degree programs concurrently or separately Terms of Reference The terms of reference identify the key issues to be addressed by the review and must address the core program evaluation criteria laid out in Section Commissioning officers may enlarge or enhance the criteria to meet the needs of their disciplines. Standard terms of reference for reviews may be found on the Vice-Provost, Academic Programs UTQAP website Announcement A review is publicly announced by the commissioning officer through appropriate unit and/or program channels and University and/or divisional media as appropriate. Submissions are invited from teaching and administrative staff, students, alumni and members of the program and/or unit community Self-study contents The degree program(s) and/or degree granting unit under review shall prepare a self-study. The self-study is a broad-based, reflective, and forward-looking report that includes critical self-analysis. It is an assessment of the strengths and challenges facing the program(s) and/or unit, the range of its activities, and the nature of its future plans. The self-study should address the terms of reference and program evaluation criteria as these will be provided to the external reviewers and will form the basis of their assessment. The process of preparing a self-study should involve faculty, students and staff. The input of others deemed to be relevant and useful, such as graduates of the program and representatives of industry, the professions, practical training programs, and employers may also be included. The involvement of these various constituencies should be outlined in the self-study. An outline of the core elements of the self-study is provided on the Vice- Provost, Academic Programs website. University of Toronto UTQAP 24

25 Figure 4: Overview of the Protocol for Cyclical Program Reviews. 1. INTERNAL UNIVERSITY PROCESS Initiation of Review by Commissioning Officer Where a review is commissioned by a Dean, the Vice-Provost, Academic Programs approves the selection of reviewers (5.7.1) Commissioning Officer announces Review, Terms of Reference and reviewers to faculty, staff, students, internal and external communities (5.6.2 and 5.6.3) Division: Self-study development; site visit scheduling (5.6.4) Commissioning Officer s sign off on Self-study External Review site visit and report (5.7.4 and 5.7.5) Response from program and Commissioning Officer Institutional Summary University accountability and reporting requirements: Committee on Academic Policy and Programs (biannual presentation) Preparation of Final Assessment Report with Implementation Plan (5.8.4) Circulation of the report and associated documents; Executive Summary posted on the Vice-Provost, Academic Programs Quality Assurance website 2. QUALITY COUNCIL PROCESS Final Assessment Report including the Implementation Plan presented to the Quality Council (5.8.6) 3. INTERNAL FOLLOW-UP PROCESS Ongoing program monitoring by the University Vice-Provost, Academic Programs may request a follow-up report from the relevant Dean to be presented to the Committee on Academic Policy and Programs. Where quality concerns are raised in the cyclical review, the Vice-Provost, Academic Programs monitors the timely implementation of improvements. University of Toronto UTQAP 25

26 In accordance with the Quality Assurance Framework, the self-study should address and document the following: a) The consistency of the program s learning outcomes with the institution s mission and divisional Degree Level Expectations, and how its graduates achieve those outcomes; b) Program-related data and measures of performance, including applicable provincial, national and professional standards (where available); c) The integrity of the data; d) Review criteria and quality indicators identified in Section below; e) Concerns and recommendations raised in previous reviews; f) Areas identified through the conduct of the self-study as requiring improvement; g) Areas that hold promise for enhancement; h) Academic services that directly contribute to the academic quality of each program under review; i) Participation of program faculty, staff and students in the self-study and how their views have been obtained and taken into account. The self-study is reviewed and approved by the commissioning officer to ensure that it meets the core elements of a self-study and program evaluation criteria Core program evaluation Reviews of undergraduate and graduate degree programs and graduate diplomas require, at minimum, the evaluation criteria set out below. Commissioning officers may enlarge or enhance the criteria to meet the needs of their disciplines. I Objectives Program is consistent with the institution s mission and unit s academic plans. Program requirements and learning outcomes are clear, appropriate and align with the relevant undergraduate and/or graduate Degree Level Expectations. II Admission requirements Admission requirements are appropriately aligned with the learning outcomes established for completion of the program. III Curriculum The curriculum reflects the current state of the discipline or area of study and is appropriate for the level of the program. Evidence of any significant innovation or creativity in the content and/or delivery of the program relative to other such programs. University of Toronto UTQAP 26

27 Mode(s) of delivery to meet the program s identified learning outcomes are appropriate and effective. IV Assessment of learning Methods for assessing student achievement of the defined learning outcomes and degree learning expectations are appropriate and effective. Appropriateness and effectiveness of the means of assessment, especially in the students final year of the program, in clearly demonstrating achievement of the program learning objectives and the program s Degree Level Expectations. V Resources Appropriateness and effectiveness of the academic unit s use of existing human, physical and financial resources in delivering its program(s). In making this assessment, reviewers must recognize the institution s autonomy in determining priorities for funding, space, and faculty allocation. VI Quality indicators Outcome measures of student performance and achievement are of particular interest. There are also important input and process measures which are known to have a strong association with quality outcomes. It is expected that many of the following listed examples will be widely used. o o o Faculty: qualifications, research and scholarly record; class sizes; percentage of classes taught by permanent or non-permanent (contractual) faculty; numbers, assignments and qualifications of part-time or temporary faculty; Students: applications and registrations; attrition rates; time-to-completion; final-year academic achievement; graduation rates; academic awards; student in-course reports on teaching; Graduates: rates of graduation; employment six months and two years after graduation; postgraduate study; "skills match"; and alumni reports on program quality when available and when permitted by the Freedom of Information and Protection of Privacy Act (FIPPA). Auditors will be instructed that these items may not be available and applicable to all programs. Assessment of the programs relative to the best of their kind offered in Canada, North America and internationally, including areas of strength and opportunities. VII Quality enhancement Initiatives taken to enhance the quality of the program and the associated learning and teaching environment. VIII Additional graduate program criteria Evidence that students time-to-completion is both monitored and managed in relation to the program s defined length and program requirements. Quality and availability of graduate supervision. University of Toronto UTQAP 27

28 Definition and application of indicators that provide evidence of faculty, student and program quality, for example: o o o Faculty: funding, honours and awards, and commitment to student mentoring; Students: grade-level for admission, scholarly output, success rates in provincial and national scholarships, competitions, awards and commitment to professional and transferable skills; Program: evidence of a program structure and faculty research that will ensure the intellectual quality of the student experience; sufficient graduate level courses that students will be able to meet the requirement that two thirds of their course requirements be met through courses at this level. vi 5.7 External evaluation: reviewer selection and review process The commissioning officer is responsible for the selection of the external review committee in consultation with the unit and/or program(s) to be reviewed. All reviewers are approved by the Office of the Provost Selection of reviewers Normally the evaluation will be conducted by a Review Committee composed of at least: 1. Two external reviewers or one internal and one external reviewer for an undergraduate program qualified by discipline and experience to review the program(s); 2. Three external reviewers or two external and one internal reviewer for a graduate program qualified by discipline and experience to review the program(s); 3. Three external reviewers or two external and one internal reviewer for the concurrent review of an undergraduate and graduate program. In cases where more than one program is being considered by the Review Committee, reviewers should be selected to ensure the appropriate review of all the programs being considered. In selecting reviewers, an appropriate balance needs to be struck between familiarity with the unit and/or program(s) under review and distance to allow for objective assessment. All members of the Review Committee must be at arm s length from the program under review, that is, they should not have a particular interest in the outcome of the review due to personal or professional relationships with members of the unit. For more detail see the Vice-Provost, Academic Programs UTQAP website. The external and institutional reviewers should be active and respected in their field. They will normally be associate or full professors with program management experience and representatives of peer institutions offering high quality programs in the field under review. Where a review is commissioned by a Dean, the Vice-Provost, Academic Programs approves the selection of reviewers. The Vice-Provost, Academic Program s UTQAP website provides further guidance on the selection of reviewers and nomination forms that set out the information that must be provided to support an informed approval process. University of Toronto UTQAP 28

29 5.7.2 Commissioning officer responsibilities The commissioning officer is responsible for ensuring that all members of the Review Committee: a) Understand their role and obligations; b) Identify and commend the program s notably strong and creative attributes; c) Describe the program s respective strengths, areas for improvement, and opportunities for enhancement; d) Recommend specific steps to be taken to improve the program, distinguishing between those the program can itself take and those that require external action; e) Recognize the institution s autonomy to determine priorities for funding, space, and faculty allocation; and f) Respect the confidentiality required for all aspects of the review process. Reviewers will be provided with clear terms of reference. The commissioning officer will also emphasize these elements when meeting with the reviewers during the course of their visit Documentation to be provided to the Review Committee The commissioning officer will identify what reports and information are to be provided to the Review Committee in advance of the site visit. Core documents that must be included are: Terms of Reference; Self-study; Previous review report including the administrative response(s); and, Any non-university commissioned reviews (for example, for professional accreditation or Ontario Council on Graduate Studies) completed since the last review of the unit and/or program. External reviewers are provided with access to all course descriptions and the curricula vitae of faculty. This can be done through provision of course calendars, web links, etc. In the case of professional programs, the views of employers and professional associations should be solicited by the unit/program and made available to the Review Committee Site visit The commissioning officer provides the site visit schedule to reviewers. Reviewers should visit together. During their visit, provision must be made for reviewers to meet with faculty, students, administrative staff and senior program administrators as well as members of relevant cognate units as determined by the commissioning officer. In the case of professional programs, the views of employers and professional associates should be made available to the reviewers. University of Toronto UTQAP 29

30 5.7.5 Review report The Review Committee submits a report normally within two months of the site visit. The Review Committee s report should address the substance of both the self-study and the evaluation criteria set out in Section above. A template for the review report should be provided to reviewers to ensure that all elements of the program appraisal are addressed. Before accepting the report as final, the commissioning officer will bring to the attention of the reviewers any clear factual errors that can be corrected in the report. The commissioning officer then formally accepts the final report and submits it to the Office of the Vice-President and Provost. 5.8 Institutional perspective and response Institutional authority: Administrative perspective The Office of the Vice-Provost, Academic Programs, as the identified institutional authority, assesses the Review Committee report. The Vice-Provost, Academic Programs requests a formal administrative response to the Review Committee report from the relevant Dean who will consult with the program and/or unit under review. The Dean responsible for the program will provide a response to the Vice-Provost, Academic Programs discussing the following: 1. The plans and recommendations proposed in the self-study; 2. The recommendations advanced by the Review Committee; and, 3. The program s response to the Review Committee s report(s). The Dean will also describe: 1. Any changes in organization, policy or governance that would be necessary to meet the recommendations; 2. The resources, financial and otherwise, that would be provided in supporting the implementation of selected recommendations; and, 3. A proposed timeline for the implementation of any of those recommendations. A timeline will be specified by the Office of the Vice-Provost, Academic Programs, outlining when the Review Committee report and Administrative Response will be brought forward to divisional and university governance Circulation of the Review Committee report The Review Committee report is a public document and should be circulated within the unit reviewed along with the Administrative Response from the Dean University accountability and reporting requirements Reviews are important mechanisms of quality assurance accountability. The accountability framework for the review of academic programs and units is contained within the Policy for Approval and Review of Academic Programs and Units (2010). The Framework outlines the following responsibilities and mechanisms: University of Toronto UTQAP 30

31 a) Program and unit reviews are considered by governance in order to allow governors to ensure that academic administrators are reviewing programs and units on a regular basis and are responding to these reviews in a manner that achieves the purpose of maintaining and improving program quality. b) The Office of the Vice-Provost, Academic Programs ensures that reviews are performed on a regular basis, that they are conducted appropriately and that the issues identified in the self-study and by reviewers are dealt with appropriately. Where quality concerns are raised in the cyclical review, the Vice-Provost, Academic Programs will monitor the timely implementation of improvements. c) Concerns may be raised in an external review report that requires a long and sustained period of response. In order to ensure that improvements are made, the Vice-Provost, Academic Programs may request a follow-up one-year report from the relevant Dean to bring forward to the Committee on Academic Policy and Programs. d) Occasionally a program may have a review or series of reviews indicating significant problems or deficiencies such that admissions to the program should be discontinued until significant improvements are made. In these situations, the divisional Dean or the Vice-Provost, Academic Programs may suspend program admissions until there is evidence that changes have been made to address quality concerns. The Office of the Vice-Provost, Academic Programs develops and submits a full and accurate Summary of the External Review Report, and the Dean s Administrative Response to the Report (excluding all confidential information) to governance through the Committee on Academic Policy and Programs of the Academic Board on a biannual basis. The Committee on Academic Policy and Programs, which reports to the Academic Board, has general responsibility for policy on, and for monitoring, the quality of education and the research activities of the University. vii The Committee s terms of reference, membership, and meeting schedule are maintained online. Its total membership is approximately 31. As with all Governing Council bodies, its membership is broadly representative of the academic divisions including Teaching Staff, Administrative Staff, Students and Alumni. viii The compendium of the summaries brought forward to each meeting is also considered by the Agenda Planning Committee of the Academic Board to determine whether there are any overall academic issues warranting discussion by the Board. The record of the discussion at AP&P is forwarded to the Executive Committee of Governing Council. At the conclusion of this governance process, the Office of the Vice-Provost, Academic Programs is responsible for preparing a Final Assessment Report which is intended as an institutional synthesis of the external evaluation and internal responses and assessments Final Assessment Report and Implementation Plan The Vice-Provost, Academic Programs compiles the Final Assessment Report providing the institutional synthesis of the external evaluation and internal responses and assessments. This report: a) Identifies significant strengths of the program; b) Identifies opportunities for program improvement and enhancement; c) Sets out and prioritizes recommendations that are selected for implementation; University of Toronto UTQAP 31

32 d) May include a confidential section (where personnel issues are required to be addressed); e) Includes an institutional Executive Summary, exclusive of any such confidential information and suitable for publication on the web; and f) Identifies an Implementation Plan including: who will be responsible for approving the recommendations set out in the Final Assessment Report; who will be responsible for providing any resources made necessary by those recommendations; who will be responsible for acting on those recommendations; timelines for acting on and monitoring the implementation of those recommendations; and g) whether a follow-up one-year report is required from the Dean. The Vice-Provost, Academic Programs provides for the timely monitoring of the implementation of the recommendations, and the appropriate distribution, including web postings, of the scheduled monitoring reports Quality Council reporting requirements The Quality Council is provided with a copy of the Final Assessment Report (excluding all confidential information) including the Implementation Plan for all completed cyclical program reviews by the Office of the Vice-Provost, Academic Programs on an annual basis Public access to review report An executive summary of the outcome of the review and subsequent implementation plan will be posted on the University s Quality Assurance website. It is left to the discretion of the program(s) and /or units themselves to decide whether or not they wish to post the full records of the review process including self-study and review report on their website. In posting any materials to do with the review all confidential materials will be removed before posting. 5.9 Quality Council audit process Auditors independently select programs for audit, typically four undergraduate and four graduate cyclical program reviews. These audits are conducted on an 8-year cycle. University of Toronto UTQAP 32

33 i ii Statement of Institutional Purpose, 1992 In 2010, the Ontario Council of Academic Vice-Presidents (OCAV) through the Ontario Universities Council on Quality Assurance (OUCQA or the Quality Council ) approved the Quality Assurance Framework for quality assurance of undergraduate and graduate programs in Ontario effective as of September The Council operates at arm s length from government to ensure its independence. iii While the QAF requires a minimum of 2/3 courses be at the graduate level, the University of Toronto requires all courses be at the graduate level. iv Proposals for new graduate diplomas and collaborative programs undergo an Expedited Approvals process (Figure 1b) without the requirement of an external appraisal (i.e., sections do not apply to these proposals). v See the Vice-Provost, Academic Programs UTQAP website for a schedule of reviews. vi While the QAF requires a minimum of 2/3 courses be at the graduate level, the University of Toronto requires all courses be at the graduate level. vii viii The Governing Council Elections Guidelines establish the manner and procedure to be used in the election of Teaching Staff, Administrative Staff, and Students to the Governing Council and Teaching Staff and Librarians to the Academic Board, including the establishment of constituencies within these categories. University of Toronto UTQAP 33