Reader David sends in this post from the Atlantic, which contains an interesting chart (copied below), ranking dozens of airline frequent flier programs in terms of award availability.

I appreciate what these kinds of rankings are trying to do, but this is unfortunately not an exact science. Let’s look at the methodology of the survey (pdf):

IdeaWorks made 6,160 booking queries at the websites of 22 frequent flier programs during February and March 2010. Travel dates spanned June through October 2010; 10 long-haul routes and 10 city pairs under 2,500 miles were checked to assess reward seat availability.
[...]
Notes regarding reward query methodology: Booking queries for a party of 2 travelers were made at frequent flier program websites during February 2010 with minor follow-up work conducted during the first week of March 2010. Some airlines require a Saturday night stay for reward travel; all of the queries used date pairings that included a Saturday night stay. While the city pairs varied for each frequent flier program, the travel dates did not. Specific dates were selected for queries and only reward seat availability for travel on the date specified was recorded; any departure time was acceptable. Furthermore, reward travel had to be available on the outbound and return dates queried. Overly circuitous routings were not allowed. When offered, online reward availability for partner airlines was always requested.

(emphasis added)

A few problems here: If you’re not comparing the same city pair, you’re not comparing apples and oranges. A sub-2500 mile itinerary could be a flight from Los Angeles to Seattle, or a flight from Inyokern, California to Harlingen, Texas. These are very different routes, with very different levels of competition (and subsequently availability).

Another issue is the fact that the survey consisted entirely of website research. Frequent flier mile ticket redemption — if you want to get the good stuff, and not just a domestic flight form Chicago to New York — is going to require a call, 99 out of 100 times. (The good stuff means complex international travel, premium cabins, and partner airlines.) Just relying on the websites is not a fair test.

Finally, what are “reward” seats? Are they the capacity-controlled “saver” awards, or are they last-seat-availability awards that cost double the miles? I’m assuming that the survey involved the former, and not the latter, but there’s no reassurance in the survey release that this was the case.

As a result, I take these results with a grain of salt. But some things aren’t surprising: Southwest has easy redemptions. Delta doesn’t. US Airways coming in dead last is a little more of a surprise, but I don’t have personal experience with their program for domestic flying.

Nonetheless, the survey is an interesting conversation starter. And the conclusions of the consultants who put it together are quite dead-on: “The revenue … comes naturally when members become enthused (and not frustrated) by program participation.” Amen.

See the chart results below.