Improving Graduate Programs at the University of Miami National University of Ireland, Galway

Transcription

1 Improving Graduate Programs at the University of Miami National University of Ireland, Galway Mary M. Sapp, Ph.D. Assistant Vice President Planning & Institutional Research University of Miami Coral Gables, Florida, U.S.A.

2 The University of Miami Private high research activity university in Coral Gables, FL Founded 1925, 11 schools (all with graduate/professional students) Programs: 115 bachelor s, 114 master s, 51 doctoral, law, & medicine 10,400 undergraduates, 3,000 graduate students, 2,000 students in law/medicine 2,350 full-time faculty, 500 parttime faculty, 10,850 total employees Budget $1.8 billion, endowment $740 million

3 The University of Miami

4 THEN: The Situation Two Years Ago Graduate education not a priority Little emphasis on analysis or management Minimal reporting (no trends, no comparability across programs, annual program review dormant) Rogue databases or missing data at program level (Continued)

5 THEN: The Situation Two Years Ago Poor quality data centrally (lots of missing & incorrect data): List of programs (differences among Graduate School, Bulletin, PIR based on active majors) Missing applied & accepted students (causes inaccurate accept rates & yields) Level (master s vs. doctoral), matriculation term, financial registration No electronic data on candidacy & errors in paper records Problems with financial support, GREs, demographics, etc.

6 Impetuses for Change National Research Council (NRC) survey to rank U.S. graduate programs started in 2006 (problems with data glaring) 10-year accreditation visit (lack of assessment of non-professional programs confirmed) New provost (hired 3 years ago), who set new priorities (Continued)

7 Impetuses for Change 2007 strategic plan (written by provost) Goals to create world-class faculty, develop prominent graduate programs, enhance undergraduate experience, & build supporting infrastructure $2.5 billion over 10 years School plans that support overall plan New Dean of Graduate School (hired 1 year ago), who is enforcing good practices

8 NOW: Culture of Analysis and Assessment Leadership support & direction Improvement in accuracy of data & processes Systematic reporting (internal & external) New analyses of peer data Systematic assessment & graduate program review Data-informed changes

9 Value of Senior Leadership and Strategic Direction Commitment at senior level sets the expectations & changes climate New provost prioritized graduate education in strategic plan & budget Strategic plan includes Goal to improve breadth & depth of graduate programs Target: half of UM s programs in top half of NRC rankings Target: increase # of doctorates to average of aspirational peers in decade (+64%) $21 million budgeted for graduate programs over 5 years New Dean of Graduate School imposing accountability & reporting

10 Number of Doctorates Granted 10-yr target 5-yr target

11 Provost s Taxonomy for Graduate Programs (from Strategic Plan) Reputation-enhancing Based on quality of students & placement in top academic positions Larger Ph.D. programs High cost (waivers & stipends, larger % fellowships, more classes) Research-enhancing Students engage in research, which supports faculty & helps attract new faculty Smaller doctoral programs (or master s) Less expensive but still net $ loss (larger % assistantships, fewer organized classes) (Continued)

12 Value of Accurate Data & Processes Revenue-enhancing Generate net revenue, subsidize other programs Usually professional or other master s Minimal student support & larger classes Each UM program asked to identify its role

13 Value of Accurate Data & Processes Using benchmark or census files (target for clean data, official reports with same totals) Asked users, analyzed current process (added codes), changed language to what programs use Sent data management reports prior to benchmark so programs clean inconsistencies & missing data Sent instructions & definitions Analyst also phoned those having problems (expanded access to online screens) Program list: now have one master list maintained by PIR (consistent with Registrar & Bulletin) Earlier problems because programs left to fend for themselves

14 New Approach to Collecting & Disseminating Graduate Data Programs enter data centrally Before benchmark PIR sends Data Management spreadsheets, listing students & highlighting potential errors so programs can clean online data After benchmark PIR sends Report version of spreadsheets List of students Remaining problems still highlighted Tables with summaries of key variables (update when errors corrected)

15 Value of Systematic Reporting Admissions analyses support market research Enrollment reports by demographics help manage Waiver & stipend summaries support reallocating resources Degrees, time to degree, graduation & retention rates monitor student progress Results of surveys of faculty (just completed) & graduate students (future) help assess climate

16 Report Spreadsheets Sent to Programs 1. Admissions: yield by characteristics; applicant data: status, test scores, UG/transfer institution data 2. Enrollments: summaries by level, full/part-time, first-time/continuing, demographics, GPA; data on these variables, contact info, dates (matriculation, master s-to-doctoral, candidacy) 3. Financial support: #, type of support, total & avg. $, central vs. department funding; data on waivers, stipends, type, funding source 4. Degrees: #, time to degree; post-graduate jobs to be entered by programs Trend over time

17 Other Reports Report Spreadsheets for 3 external surveys: Council of Graduate Schools Peterson s U.S. National Science Foundation Graduation & retention rates Survey results

18 Value of Referencing Peer Data NRC ( preliminary analysis of 30 peers Academic Analytics ( faculty are key to graduate education U.S. News ( trustees asked for explanation of change in program rankings U.S. Department of Education ( UM high diversity U.S. National Science Foundation ( research, surveys of graduate students Carnegie classifications ( Internal dashboards & other reports (peers, trends, comparisons across departments)

19 Data Used for NRC Rankings Student enrollment International students Female enrollment Under-represented Minority enrollment GRE score Student financing Student treatment variables Number of Ph.D.s (5 years) Median time to degree Student placement Number of faculty Female faculty Under-represented minority faculty Faculty Publications Faculty Citations (except humanities) Faculty Grants Completion rates (6 & 8 years) Faculty Honors & awards

20 Data Collected by Academic Analytics Journal publications Citations Book publications Federal research grants Awards & honors Overall rankings of program faculty (weighted average) % of faculty with each & per faculty

21 Dashboard

22 Dashboard

23 Dashboard

24 Value of Assessing Effectiveness of Graduate Education & Program Review Regional accreditor requires the following: Mission statement (tied to university mission) Student learning objectives (at least 3; should have targets) Measures for assessing if objectives are being met (at least 2 per objective) Summary of data from most recent year s assessment Statement of how program was changed in light of data Program reviews (8 this year, 7 next fall)

25 Graduate Program Review Programs write self-assessment & provide key data Internal subcommittee of graduate faculty External review committee (2-day campus visit) Graduate Council discusses 3 reports & sends recommendations to Provost & Faculty Senate Goal: Continuous Program Improvement

26 Program Self Assessment Five-year strategic plan Strengths & weaknesses Comparisons to 3 peers & 3 aspirationals Plan to become ranked in top half of discipline

27 Data for Graduate Program Review Program size Student financial support Number of applications & yield GRE Scores, other test scores, incoming GPAs Number of degrees granted Time to completion Placement 5-year trends

28 Assessment Rubric Rating Scale and Explanations 1 = Unacceptable Error(s) in exposition of the field and/or omission of key source(s) 2 = Poor Minor errors, omissions, and/or lack of synthesis 3 = Average/ Acceptable Adequate and accurate exposition of key sources 4 = Very Good Good coverage and synthesis of key sources plus additional relevant material 5 = Exceptional Thorough review and excellent synthesis of sources, including some obscure but relevant ones Appropriate methodology Errors in methodology selection and/or use Minor methodological errors and/or omissions Methodology applied correctly and adequately; appropriate documentation Methodology applied correctly, explained clearly, and documented well Mastery of finer points of methodology plus elegant application and/or supplementary approaches Application of knowledge and methodology to original research topic Discipline and methodology not referenced/applied well Some links to knowledge and methodology of discipline but not clearly integrated with research Adequate connection between knowledge of discipline and use of methodology and research Clear exposition of relationship of disciplinary knowledge and methodology to original research Insightful references to sources and application of methodology to excellent research topic Critical thinking Muddled presentation with errors in reasoning and/or without much analysis and synthesis Reasoning sometimes confused, simplistic, and/or not clearly explained Adequate reasoning, explanation of assumptions, and supporting evidence Clear reasoning with organized presentation of evidence, assumptions, and conclusions Clear and organized argument that represents sound, original, and complex thought Effective written communication Writing generally unclear, with consistent errors and/or poor organization Writing sometimes unclear with weak organization and/or grammatical errors Writing clear, concise, and organized, with minor or no grammatical errors Writing generally error-free with clear organization and depth Elegant writing with fully developed arguments, clear organization, and correct grammar Effective oral communication Presentation generally unclear, with poor organization and/or marred by distracting mannerisms or language Presentation sometimes unclear, with weak organization, and/or some distracting mannerisms or language Presentation organized to convey main points of thesis/dissertation clearly and without distractions Articulate presentation with clear organization and professional language Elegant, confident, and engaging presentation with clear organization and flow Overall quality (not necessarily average of earlier ratings) Unacceptable Poor Average/acceptable Very Good Exceptional Knowledge of the discipline Rating (1-5, from scale to left)

29 Value of Data-Informed Changes Good data, effective analyses, & wise decision makers create good decisions & improvement Examples at UM: Changes in student support (tuition waivers, stipends, health insurance, travel & research funds) Analysis of strength of programs (will affect future decisions about resource allocation) Reallocation of responsibilities (administrative support, which had been decentralized 2 decades ago, will be more centralized) Data shouldn t drive decisions but should inform them.

30 UM has come a long way in under 2 years! I hope this inspires you to do the same.