Thanks scarberiankhatru and everyone else.
I notice Scarborough, between 81 - 91, lost 200k worth of its population, while Etobicoke + North York each gained 100-150k. Could this have been simply due to a change in the city limits at the time? That would be ample evidence to explain this...