Go to the head of the class

I can ‘t help myself. I am drawn to lists of the best and the worst.  The best colleges and universities, The most livable cities.  The worst cars.   The question this blog entry pushes me to think about are the factors that determine the ranking. The trigger was a July 1, 2022 article in the Washington Post titled “Columbia to skip U.S. News rankings after professor questioned data”.

The professor is Michael Thaddeus, and it is hard to brush him aside as sour grapes since Columbia has consistently ranked in the top schools. The article also quotes the president of Princeton University which has topped the list in many years.  He has written that rankings are a problem because they produce “damaging incentives.”.  He goes further, calling the rankings “a slightly daft obsession”.

Here’s my list of seven reasons daftness may apply.

1. It is hard to reduce the complexity and individuality of colleges and universities, cities, or anything else ranked to the numbers used in a way that is comparable entry by entry. Greatness is not always measured well by linear ranking. (If you want more on that hit results1st.org website and read “Assumptions for Greatness”) 
2. The factors used to create points may not reflect the most important elements of a group or product ranked.  In education, small class size is not helpful with a bad teacher.  And the number of parks in a city may not matter if a person without a car cannot get there with public transportation. 
3. Rankings are based on objective factors that translate across those rated.  They cannot include the idiosyncrasies that may matter most—such as level of pride in a city or the way a school builds engagement at an interpersonal level. 
4. Groups and other entities rated often think they have no choice but to focus on what will improve their place in the ratings if that is seen as bringing us customers.  This forms an external beacon that can dominate strategic direction.
5. The whole obscures the parts.  Most people don’t live just in a city. They also live in a neighborhood. Social supports and fabric have a strong local component.  Similarly, the math department at a university may have a very different feel than the English department. Even if that level of data is or could be available, most of us want to boil it down to the one list.
6. The differences between entries next to each other may be too small to have any practical significance.  The top or bottom five might well be treated equally.
7. When used locally many ratings are popularity contests.  Most papers publish a “best list” for restaurants, dog groomers or any other good or service. The best receive the most votes, often influenced by internet marketing and with no clarity as to what best means to any of the voters.  This drags down the standard.

So, what should I do?  First, take those ratings with two grains of salt. Enjoy them with amusement as much as best guidance. Look for large differences rather than small one.  Seek to see variation among factors, including the weightings often used to make some metrics more important than others. I have more faith in rating systems such as Consumer Reports ratings of new and used cars. They publish the results by factor before giving an overall score. If I think reliability is key, I go more by rate of repair comparisons. If comfort and convenience is critical, I look at those rankings. The same for crash protection, fuel and emissions and other factors.  I like being ab le to interact with the metrics based on what I find important.

And where you might base a decision on rankings or ratings please set them aside and ask yourself what factors are important to you.  If these do not show up in the ratings, be careful!

Now back to the U.S. News and World Report ranking of colleges.  I want to see how well my alma matter was rated.  Rank has its privileges.