Fewer children isn't something that you can easily point to as having a clear affect on places at the neighbourhood scale, or even (sometimes) the municipal scale. There might be neighbourhoods in the old city of Toronto old enough to be home to a diverse enough set of age cohorts for there to be a gradual decline in kids per house over time (as opposed to some suburban areas perched on the abyss of a mini demographic meltdown because almost everyone's the same age and the kids leave at the same time), but gentrification is probably doing more to these old Toronto places than changing birth rates. It would be a different case if there were no rooming/subdivided houses, but the reality is these chopped up and rented houses do make a difference when converted back into one-family homes, going from 8 to 3 residents, or whatever: if a street has 100 stable residents, that one gentrified house just caused a 5% population drop.
If memory serves, the old city of Toronto lost about 1,000 people between 2001 and 2006. Of course, there's the usual disclaimers about census undercounts, but I suspect the sheer volume of new construction since 2006 has probably tipped the old city back towards growth, even if the average condo is only inhabited by 1.2 people or 1.4 people or something like that.
Also, if memory serves, Etobicoke, East York, and York lost people between 2001 and 2006, with York now slightly smaller than it was in 1971. It's frustrating trying to talk about these places and figures in meaningful, contemporary ways since the next census is so close.
But even booming Milton actually lost people between 1996 and 2001, and then went on to grow by 71% by 2006, so don't take these ups and downs at face value.