Russell Beattie comments on the NPR story that there are many empty seats at Athens. In fact, so desperate is the IOC to fill seats (looks bad on TV, man) that they are thinking of giving it away.
"IOC officials, worried by the television images being flashed around the world of athletes competing in near empty stadiums, have told the Athens Games organisers to give tickets away for free if necessary." -news24.com
Maybe people were not attending because they were not wearing the right logos on their t-shirts.