I think you might be on to something here.
I don't think "throwing out all the low rankings" is the right answer, but I do think that setting the "rank+" line at 5 might be a good idea (i.e., the categories go from 1 to 5+ instead of 10+).
Several reasons for this--
1. Nearly all applicants rank at least 4-5 programs, but after that the spread gets really big, with some ranking 5 to 6 and some ranking 18-20. A 6 ranking for the former person is kind of the same as a 20 ranking for the latter (i.e., both last) but each would affect an average very differently.
2. Quite a few people who submit their lists add a "no particular order" addendum to their list, usually after 6-10 but never before 5. Changing the final category to 5+ would "equalize" these discrepancies in reporting ROLs.
3. The lower ranks would be appropriately diminished in their contribution to an average, while maintaining the accuracy of keeping the top numbers actual.
4. As you mentioned above, the top of people's rank lists are probably scrutinized by the applicants a lot more--there's not too much information to be gathered from someone ranking a program 7th on a list of 8 vs 16th on a list of 20.
This would be a big change that would obliterate some data, so I want to get some feedback before I make this an actual change. Here is what the data would look like if this change was made:
https://docs.google.com/spreadsheet/ccc?key=0AnzZUifXW_SgdGtmQmFWMXZjb21kd253UklCNFJkQnc#gid=0
I admit I don't think the data loses much and it might be a big improvement. Thoughts?
Ian