I think there's more to it than aesthetics. An expressway doesn't simply influence the land it occupies directly, but also its periphery. Since it is, by definition, high-speed, it effectively prevents any sort of direct interaction with the land around it, except of course at the off-ramps, which occupy a massive amount of land (although the circular offramp that feeds York St. encircles a rather charming little park).
Granted, developers are starting to put up condos that are so close to the Gardiner that you can (possibly literally) step off your balcony onto the highway; however, no matter how much you cram around it, the Gardiner necessarily precludes the type of development that would unify the lands on either side. Effectively, it bisects the neighbourhood much like the Spadina Expressway would have bisected the Annex (not that I'm comparing the legacy of the Annex to the waterfront property... simply that an expressway exerts a sphere of influence that is greater than its footprint).
In this sense, it doesn't act as a barrier so much that you have to climb over it to get to the other side, but it prevents the slivers of land on either side from becoming a cohesive whole. A boulevard, if executed properly of course, could potentially resolve that problem (or maybe not... I'm no traffic analyst).