Wednesday, August 16, 2006

Ranking Colleges

It's apparently that time of the year when U.S. News and World Report releases its annual ranking of U.S. colleges. There is little doubt that the rankings are very dubious, although that hasn't stopped them from deeply influencing the choice of universities for a great many students (besides strongly influencing the marketing material churned out by a good number of these universities).

Some people appear to generalize from this state of affairs to claim that ranking universities meaningfully is fundamentally impossible. David Leonhardt in the New York Times rightly debunks this view with counter-arguments that point out the utility of ranking in other equivalent spheres (e.g., students are graded all the time based on test results that may have little to do with their abilities).

One effect to keep in mind in all this, however, is the non-linear nature of the bias induced by rankings, and the socially sub-optimal consequences that may ensue.

Here is how this works: A key determinant of student performance is the level of performance of his/her peers. Many of the benefits of studying at a top school emerge from the presence of other smart students at the same school -- be it due to a higher level of competition, the cultivation of "higher quality" social networks, or the non-linear improvements from collaboration. Therefore, the definition of a "good" school is partially recursive: A "good" school is one that attracts "good" students; "good" students are those that go to "good" schools.

In such a recursive system, bad rankings can significantly distort the assignment of students to schools. Imagine the rankings rate an "innately bad" school higher than an "innately good" one. Now, innately "good" students are going to drift towards the higher-rated school, causing an increase in the overall quality of the "innately bad" school and a decrease in the overall quality of the "innately good" school. This effect feeds back into providing an even better ranking for the "innately bad" school the next year. Which means that even more "good" students will gravitate towards it the next year! And so on...