That is a pretty empty argument.
You could have made the same case for not allowing any new construction to begin downtown many years ago as the Gardiner has always been packed, at least for the last 20 years that I remember it. But we did build. More people than ever come downtown. More people than ever want to live downtown. And communters have found other ways to get into the city or people have moved to other means such as public transit. Also, I'd argue that the east-west danforth subway line is hardly full as you imply. Yonge subway yes - Danforth no. So your argument is quite moot.
Also, taking down the Gardiner does not mean people need to lose their ability to get downtown. Options include wide boulevards with lower speed limits ( since the city core should be a destination and not a place you drive past on an expressway ). Or if you want to really think big, the alternate option would be to build a tunnel for the through traffic so that you retain an express while moving local cars to actual streets and boulevards.
Likely the best option is a tunnel and boulevard combination - but that will take a lot of money, so who knows when and if it will happen.
I had the opportunity to visit Seoul and Tokyo recently. In addition to outstanding transit systems both mega cities had urban thoroughfares, very wide streets with 6+ lanes, where vehicles could go fast (60km+). In Tokyo there are multiple highways that wind through the core of the city.
When I consider arguments to remove the Gardiner, these thoughts come to mind:
1. The Gardiner-DVP is the only expressway into Toronto's core.
2. Our core was built years ago and does not have a network of wide urban thoroughfares, either east/west or north/south that can accommodate large volumes of traffic. Our narrow main streets (Bloor, College, Dundas, Queen, King, Dufferin, Bathurst, Bay, Yonge, Jarvis, etc, etc) are a beautiful feature of our City.
3. We have a limited public transit network (with virtually no redundancy).
The Gardiner is an essential part of our infrastructure. Perhaps as we build out the DRL and other transit projects it will become less essential. Until then, the argument to remove it would strand many commuters and make our City much less efficient.