Academy Rankings Tell You a Lot, But Not Who's No. 1 in Any Field

Jeffrey Mervis
Science 1 October 2010: 18-19. 

This week's release of the long-awaited assessment of the quality of U.S. research doctoral programs by the National Academies' National Research Council will disappoint those who simply want to know who's No. 1 in any particular field, because the NRC assessment can look quite different depending on your definition of "best." 

 This National Academies' NRC study was discussed in a recent meeting I attended of the UMass Amherst Faculty Senate's council which is the liaison to the University's Development Office.  Rankings of this kind are always of interest to University administrations, naturally. Everyone speaking about it acknowledged that the 'rankings' are problematic in a number of ways.  One member commented that the people doing the ranking wouldn't have first-hand knowledge of the  departments they were judging among - it's a matter of reputation, apparently, rather than anything based on real data. That said, people will make use of the numbers in whatever way will benefit their agenda.  From the article in Science:

To be sure, NRC does rank programs—but oh so carefully. Instead of assigning a single score to each program in a particular field, the assessment ranks the program on five different scales. Each score is also presented as a range of rankings reflecting the 5th and 95th percentiles of the scores it received. The scales themselves are based on 20 characteristics (see table, p. 19) that the NRC panel deemed appropriate for a quantitative assessment. Two are supposed to portray the overall quality of the program—one derived from a reputational survey (the R scale), the other from a quantitative analysis (the S scale). Three others rely on subsets that address important dimensions of quality: research activity, student support and outcomes, and diversity. The report itself highlights the uncertainties generated by such an exercise by calling the results "illustrative rankings [that] are neither endorsed nor recommended by the NRC as an authoritative conclusion about the relative quality of doctoral programs."

 The report's Excel spreadsheets are available at www.PhDs.org or www.nap.edu/rdp.