An Analysis of the NRC's Assessment of the Doctoral Programs in Public Affairs Göktuğ Morçöl & Sehee Han Pennsylvania State University Prepared for the NASPAA Annual Conference November 2014, Albuquerque, NM
Background Information We conducted analyses on the National Research Council s (NRC) Report on PhD Programs in Public Affairs (http://sites.nationalacademies.org/pga/resdoc/). We extracted the data from the NRC spreadsheet (http://www.nap.edu/rdp/). Background Information on the NRC Study: The NRC studied 5004 doctoral programs in various fields at 212 universities. The data were gathered in 2005 and 2006. The report was published in 2010. In the Public Affairs category, there were 54 programs.
Next NRC study on PhD Programs? There is a possibility that the NRC will conduct another study in the coming years. there are some preliminary conversations within the NRC, and with our partners, about whether and how to conduct this study, but there are no firm plans on the table at the moment. (email message from a National Research Council representative)
Three Categories of Variables Used in the NRC Rankings Faculty productivity: Publishing patterns Research funding Awards for scholarship Student characteristics: Student support Completion rates Diversity of the academic environment Diversity among faculty and students (Source: Jeremiah P. Ostriker, Paul W. Holland, Charlotte V. Kuh, & James A. Voytuk (Eds.), A Revised Guide to the Methodology of the Data-Based Assessment of Research-Doctorate Programs in the United States (2010); Committee to Assess Research (http://www.nap.edu/catalog/12974.html).
Types of Rankings in the NRC Report Survey-based rankings (S Rankings) Regression-based rankings (R Rankings) Separate rankings for the three dimensions of program quality: Research activity Student support and outcomes Diversity of academic environment
S an R Rankings in the NRC Report S RANKINGS: Based on a survey among faculty members at different institutions. They were asked to: Weight of (assigned importance to) 21 characteristics that the study committee determined to be factors contributing to program quality. The weights of characteristics vary by field based on faculty survey responses in each of those fields. R RANKINGS: An index of the 21 program quality variables based on the weights calculated from faculty ratings of a sample of programs in their field. Multiple regression and principal components analyses were used to develop the index scores. (For more details, see the slides on methodology at the end.)
Question: What are the most important factors contributing to the NRC s S and R rankings? The following tables and charts display the 5 th percentile rankings of the programs: Their best rankings after the top 5% of the 500 simulations were removed. (For more information about how the NRC calculated the percentile rankings, see the slides on methodology at the end.)
Most important factors contributing to S and R Rankings S RANKINGS Pearson Spearman 1. Research Activity (5th Q) 0.92 0.93 2. Average # of Publications Per Faculty -0.76-0.78 3. % Faculty w Grants -0.74-0.74 4. Average GRE scores -0.68-0.70 5. Average Citations per Publication -0.66-0.72 R RANKINGS Pearson Spearman 1. Average GRE Scores -0.73-0.72 2. % International Students -0.63-0.67 3. Research Activity (5th Quartile) 0.63 0.62 4. Is student work space provided? -0.54-0.55 5. Average # of Publications Per Faculty -0.50-0.56 (Color codes in the tables: Red: Student-related variable ;Black Faculty-related variable) Pearson and Spearman correlations are very similar. The biggest contributors to S rankings are faculty-related factors. Both student-related and faculty-related factors contributed to R rankings.
Question: How do the most important faculty-related and student-related factors relate to the R rankings?
Top Two Faculty-Related Factors & R Rankings: 1. Research Activity (Cubic is the best fitting line.) Research activity does not pay off for some highly productive programs.
Top Two Faculty-Related Factors & R Rankings: 2. Faculty Publications (Quadratic is the best fitting line.) Faculty publications do not pay off for some highly productive programs.
Top Two Student-Related Factors & R Rankings: GRE Scores (Linear is the best fitting line.) GRE scores are linearly related to rankings.
Top Two Student-Related Factors & R Rankings: International Students (Cubic is the best fitting line.) Some highly ranked programs have smaller percentages of international students!?
Questions: Are there regional differences in R rankings? Is there a difference between public and private institutions in their R rankings?
Differences among the Regions of US in R Rankings (sig. of F=.161)
R Rankings of Public vs. Private Universities (sig. of T=.018)
Question: Are there similarities between the NRC (doctoral) R rankings and the US News & World Report rankings of master s programs?
NRC Doctoral Rankings and US News Master s Degree Rankings NRC R Rank NRC S Rank US News Rank 2014 Pearson Spearman Pearson Spearman Pearson Spearman US News Average Assessment Score in 2007 US News Rank of Public Affairs Master's Programs in 2007 -.573 ** -.613 ** -.447 * -.467 ** (n=31) (n=31) (n=31) (n=31).568 **.613 **.379 *.467 ** 0.322 0.813 (n=31) (n=31) (n=31) (n=31) (n=31) (n=31) US News Rank of Public Affairs Master's Programs in 2014.787 **.798 **.665 **.670 ** (n=51) (n=51) (n=51) (n=51) ** Correlation is significant at the 0.01 level (2-tailed). * Correlation is significant at the 0.05 level (2-tailed). NRC and US News rankings are correlated. Spearman correlations are higher. US News rankings are consistent over the years.
US News Rankings of Master s Programs (2014) and NRC Rankings of PhD Programs (2005) NRC (2005) and US News (2014) rankings are quite linearly related.
Conclusions Both faculty productivity and student characteristics matter in the NRC rankings. Faculty productivity contributes more to survey-based (S) rankings. Faculty members seem to be actually rating their colleagues productivity when they rate other programs. NRC report notes: Research activity is the dimensional measure that most closely tracks the overall measures of program quality, because in all fields, both the survey-based or direct measure based on abstract faculty preferences and the regressionbased measure also puts high weight on the measures of research productivity in addition to the measure of program size. (Source: A Revised Guide to the Methodology of the Data-Based Assessment of Research- DoctoratePrograms in the United States; http://www.nap.edu/catalog/12974.html)
Conclusions Private universities rank significantly higher than public universities. NRC rankings of doctoral programs are highly correlated with US News rankings of master s programs.
Thank you.
The following slides are about the methodology of the NRC study.
Categories of variables that were weighed by survey participants
Explanation of percentile rankings (S and R rankings) For every program variable, two random values are generated one for the data value and one for the weight. The product of these summed across the 21 variables is then used to calculate a rating, which is compared with other program ratings to get a ranking. The uncertainty in program rankings is quantified, in part, by calculating the S Ranking and R Ranking, respectively, of a given program 500 times, each time with a different and randomly selected half-sample of respondents. The resulting 500 rankings are numerically ordered and the lowest and highest five percent are excluded. The 5th and 95th percentile rankings in the ordered list of 500 define the range of rankings are shown in the tables.
Explanation of percentile rankings (direct quotes from the NRC report) Because of the various sources of uncertainty, which are discussed at greater length in Appendix A, each ranking is expressed as a range of values. These ranges were obtained by taking into account the different sources of uncertainty in these ratings (statistical variability from the estimation, program data variability, and variability among raters). The measure of uncertainty is expressed by reporting the end points of a range that includes 90 percent of all the ratings for a program. These are the 5th percentile point and the 95th percentile point. We obtain both the survey-based weights and coefficients from regressions through calculations carried out 500 times, each time with a different randomly chosen set of faculty, to generate a distribution of ratings that reflects their uncertainties. For both the S and the R rankings, we obtain the range of rankings for each program by trimming the bottom five percent and the top five percent of the 500 rankings to obtain the range that includes 90 percent of the program s rankings. This method of calculating ratings and rankings takes into account variability in rater assessment of what contributes to program quality within a field, variability in values of the measures for a particular program, and the range of error in the statistical estimation. It is important to note that these techniques give us a range of rankings for most programs. We do not know the exact ranking for each program, and to try to obtain one by averaging, for example could be misleading, because we have not imposed any particular distribution on the range of rankings. (Source: A Revised Guide to the Methodology of the Data-Based Assessment of Research-Doctorate Programs in the United States (2010) http://www.nap.edu/catalog/12974.html, pp. 17-18)
Summary of the methods used in calculating the S and R rankings
A more detailed view of methods of calculating R and S rankings
An even more detailed view of the methods of calculating R and S rankings
An example of calculations of R ratings (Source: Revised methodology guide, p. 22)
An example of calculations of R ratings (Source: Revised methodology guide, p. 22)