I know the eye has been on Old City Hall for this, but p'haps The Museum Of Toronto could be fit in neatly. Or a branch of the National Gallery or Vancouver Art Gallery for example.
Richard White's mention of a big IMAX theatre has some appeal - though that would entail some heavy alterations. Some more big-screen theatres downtown would definitely be a plus. If that route was taken, there's no reason why a small multi-purpose theatrical space couldn't be included. Well, hell, the possibilities are pretty vast.
Basically, instead of simply going for more retail, it'd be interesting to see if that space couldn't also become something more variegated in use, more cultural and social - and interesting.