As Ramako says, we're one of those folks who bought near the Gardiner for reasons of access. As for noise, I'd argue that Lake Shore is just as noisy. If the Gardiner were to come down, what would replace it? It is ugly though, and walking underneath it to get to/from the waterfront/rest of the city isn't pleasant.
The Gardiner is way less of a barrier between downtown and the lake than the rail lands. It was put there for a reason which pretty much still exists. It makes more sense to me to open the land around and under it to other uses. Of course it would be nicer buried, but it would also be nicer if we hid it more or (gasp!) decorated it.
The drive on the Gardner through those towering condos is MAGNIFICENT!