I'm sure that if there was a crosstown elevated freeway currently located between St.Clair and Eglinton people would be saying that there is no way you could take it down and there would need to be a replacement.
The reality is that the current freeway probably doesn't work as well as it could if the freeway ended in the Strachan-Bathurst area with ramps that got people on and off Bathurst northward, Front eastward, and Lakeshore eastward without the need to turn. A lot of the congestion in the downtown is caused by people trying to get on the Gardiner through a series of turns at stoplights. If you could go straight on Front and get on the Gardiner, take one-way Richmond/Adelaide to Bathurst and then head south to get on the Gardiner, and take Lakeshore to get on the Gardiner and completely avoid Bay and Spadina things would move better.
Since it already exists they should just spruce it up. At some point even suburbanites will not care that it is taken down because GO's service will be far better than the alternative of going onto the Gardiner which as the downtown population grows will be that much more useless as a transportation option. That time hasn't come yet, and GO's service isn't there yet... but eventually it will be.