I count 17 towers on this map and 13 of them are vapourware.
Far more actual growth is happening along the waterfront than north of Queen. There are 1500 units under construction at Quayside alone. The Queen location is a wash for West Don Lands; for other waterfront neighborhoods it's farther and much less pleasant to walk to. Meanwhile Regent Park is already getting a new library which should be large.
The biggest problem though is the specific site. This "District Library" is intended to be a community hub that can draw people for a variety of activities. That works best near parks and/or other public facilities, and near a lot of housing. That was the logic behind the previous two sites for this building, and for the existing 1970s location in St. Lawrence Neighbourhood. That is absolutely how public facilities should be planned, to coexist and support each other.
The Queen Street site has none of those qualities. There's no adjacent public space. The sidewalks are tight. Parliament is a traffic sewer and the DVP-Richmond ramp is adjacent. Queen Street here has little foot traffic.
If you're going to serve many people and create an attractive place for public gathering, this is not the spot you would choose.
This is the major library project for this part of Toronto in the next two generations and it's going in the wrong place.