I walk in the core very frequently, and yes, there are a LOT of buildings under 3 storeys tall, if by the core we mean between Spadina Ave and Jarvis st. Don't believe me? Let me show you
Between University and Spadina
Huron St
http://goo.gl/maps/ozgVf - this particularly will make you think you are in US midwest, not downtown Toronto, honestly...
D'arcy St
http://goo.gl/maps/PaX1R
Grandge Pl
http://goo.gl/maps/L1PsV
Beverly st
http://goo.gl/maps/DpV3Z
Between Yonge and Jarvis
McGill St
http://goo.gl/maps/9k1om
Maitland st
http://goo.gl/maps/u5jcg
Dundonald st
http://goo.gl/maps/lL7x6
I find the area between College/Queen west of University and between Gerrard/Wellesley east of Yonge has the largest stock of low rises houses. These should target higher density, after south of Queen and Yonge st are developed.