I disagree regarding the diversification of employment downtown. I should reword that, you're right, the FIRE industry dominates the core, but, overall the last 10 years that dominance hasn't been increasing, as other industries have been growing.
So if you're going back far enough when there was much more industry downtown, you're indeed correct, there was a much wider range of jobs. But lets just talk about the last 10 years or so. More jobs of surfaced in the technology industry (most related to social media / marketing), actually a recent report was conducting stating Toronto has the 3rd most start ups in this sector in North America (and that number is growing).
I completely agree that the rest of the city suffers, where employment levels have slowly fallen (or remained stagnant) over the last 20 year or so. There are no signs this will change and probably very little that could actually change this at this point. Yes the core may continue to grow but I think the outside of that, not only will Toronto keep losing jobs but those areas will lose more middle class residents as we have been seeing.