I think people just don't realize what's going to happen. Imagine University Avenue, but with a 70 km/h speed limit (University's is 50) and double the traffic volume (University gets 40,000-50,000 cars per day). That's exactly what would be replacing the Gardiner - a wall of slow-moving cars during peak hours, and a wall of fast-moving cars the rest of the day.
Reclaiming space is nice, but it's a lot nicer to keep cars on the Gardiner separated from local traffic. What the city's doing right now makes a lot of sense - reduce the Gardiner's footprint as much as possible without significantly disrupting traffic, which includes pedestrian and bike traffic that currently doesn't have to cross an eight-lane, high-speed surface road.