I would disagree with the premise that the Allen road section is a "worse" route than an alternative like Dufferin or Bathurst. It's lower ridership than Yonge, for sure, but everywhere in Toronto is.
If you compare the Spadina line's underground stations (e.g. St.Clair West -> St. Andrew) to the Allen road sections (Wilson->Eglinton West), the difference in ridership between stations on the Yonge and Spadina lines is about the same regardless of whether you look at the Allen section.
Even on the Yonge line, and even in places like Yonge & Eglinton which seem very dense and urban, ridership is mostly a function of feeder buses. There's really no station outside of downtown which generates huge amounts of walk-in traffic. So, overall the Allen isn't as big of a hamper as some people think. Even as a pedestrian, the station and overpass design of the Allen is actually fairly decent.
I'd guess that, if the Allen had been built along Bathurst or Dufferin, you'd have seen <10% more ridership over what we have today. Maybe less, since stations like Glencairn wouldn't have been economical as underground stations (like on the Yonge line). The tradeoff for slightly lower ridership was a major reduction in costs.