Just so you're all clear ... this isn't unique to Toronto at all - you'll see it in most major cities - even across Canada! That is some people who live out of the city are happy to never venture down (or up / or whatever the direction may be) other then the rare sporting event / theatre / so on ... I know a lot of people who live in the suburbs of Vancouver who say this all the time! It's to busy / dirty / people are rude (yes, even in Vancouver I've been told this) it's all relative of course - if you were to ask them people in Toronto are the rudest
Anyway this is all in good fun - I think what gets to a lot of people is the anti-Toronto sentiment that exists on comment boards for the major newspapers in this city (there's one in particular - and no it's not the sun ...) also, whenever something negative occurs - say every region in the GTA increased tax, typically with Toronto increasing it the least (which is a bad thing, but a completely other story) you'll see the typical outraged Toronto citizen shot.
Now go on the streets and talk to people / friends / what not - god everyone I know who lives around my area / downtown / other central locations absolutely love it - and typically think others are crazy for not sharing that sentiment. Yea we our complaints about traffic / TTC / road conditions - but in general most people I know who live in the city, really do love it (in central areas at least, I don't know about others - well, even if I do, this is a central vs suburban thing so there comments aren't really important as they fit in that later camp ... which is fine!