I think seeing income levels mapped out by area is a tad misleading.
There are pockets in the "upside down T" that are lower income. Some of it by design, as even lower-income individuals (such as recent immigrants, like my parents once were) need somewhere to rent. All those apartment blocks that we built in the post-war period are perfect for that.
Outside of the "upside down T", I believe the way income is mapped out is affected by housing built-form and types. In the upper left and right corners of the city, single-detached homes rule the geography and are more dispersed and on larger lots than within the 'T'. This means lower density of middle-class homeowners and the income when mapped geographically, skews towards those dwelling in the dense suburban apartment blocks.
Add in the generally higher desirability of neighbourhoods that can support retail strips like Corso Italia in denser, less auto-dependent residential neighbourhoods compared to say, Bridlewood in Scarborough, and even the single-detached homeowners income logically begin trending in certain direction, despite most house-owners in both areas being upper-middle class and sleeping on (multi-)million-dollar homes.
My point is, I think the built-form of the neighbourhoods are leading to both the disparities in income levels, and how it looks mapped out (potentially painting a more severe narrative than in reality).