They did it by actually buying a ticket on every airline in their test. (They only included airlines that HAD a guarantee, naturally, so that meant American, Continental, Delta (sort of), Northwest, and United. US Airways, Southwest, AirTran, jetBlue, etc., weren’t included, since they don’t have a guarantee.)
Because FareCompare’s fare alerts — which I have strongly recommended in the past — give you several hours’ advance warning when a fare is about to drop, they knew exactly which tickets to buy. They bought the tickets before the fare drop went live. When the fare went down, they took a screenshot of a lower fare and filed for a refund and/or voucher with the airline.
What they found: Lots of variation. Each airline eventually came through, but the amount of effort required varied greatly. It wasn’t always easy: Some denied refund requests at first, or didn’t respond within 24 hours.
The airlines’ policies vary, too. Most required a $5 difference before considering a refund, but Continental required $10. Most give a cash refund, but United only gives vouchers. Most accept a lower fare published on any site, including their own, while American and Northwest bizarrely exclude lower fares that appear on their own sites. Delta doesn’t have a guarantee, per se, but they’ll refund your ticket within 24 hours.
It’s a great experiment. Go read the whole thing.
Note that FareCompare was testing the airlines’ sites only. Some online travel agencies have guarantees as well. For example, the folks at Peter Greenberg’s site recently had to step in to help a reader enforce Expedia’s guarantee.
In all these cases, it’s up to the customer to proactively search for a lower price within 24 hours. No one is going to volunteer the news that the price has dropped. But if you’re willing to spend the time and effort to check the prices again and wrestle with customer service, you could collect a few bucks.