I'm not sure if this is an American or Texan thing. I spent a month in Boston in 2004 and while I was there, no one seemed to have any difficulty understanding me, or even comment that I had an accent (which I still get at least once a week in Texas). So I don't know what to think if I'm honest.
I don't think I'm helped by British tv shows either. When I say this, I'm thinking of costume dramas and murder mysteries (like Miss Marple) in which the character speak with a particular English accent that only certain types well educated middle class people speak. I speak the same sort of English, but not in the same accent, which confuses people I think.
Don't get me wrong, there are exceptions. I was watching a British show about spies on cable a few days ago. The story saw this poor (and I mean financially) black kid breaking into a car and accidentally stealing a high tech computer. Whenever he or his family came on the screen, they were subtitled, despite the fact he was speaking English and I had no difficulty understanding them.
Anyway, I said all that to say this. I think that my accent might be changing ever so slightly. In the last couple of years I've had more and more people ask me if I'm Australian. It's still only running at one or two people a month asking me that, but to my mind my accent sounds nothing like theirs so I catch myself thinking "What the heck makes you think that?" a lot when I'm asked that.
On a related note they have a student run news station at TCU and for a while they had a South African or Zimbabwean presenter on it and a couple of people used told me she had a "very correct English accent," which confused me because she sounded no more English than Americans do.
Finally on the subject of accents, I was in St. Louis a while ago and someone asked me where I was from.
When I said he proceeded to ask me if they had accents in England, because I didn't sound like I was speaking the English he heard on tv. Trust me when I say that that was another "What the heck makes you think that?" moment.