Again, not that I prioritize this.........but its not as limited a choice as you think.
Paris built their Olympic village about 8km from the City Centre (outside Paris, proper's limits)
View attachment 573912
Downsview aside, you pretty much need ~40 hectares or 100 acres for the athletes village.
Any golf course that's located more than 50% on table land is an obvious choice.
Two in the City would meet that criteria, Downsview Dells and Islington.
Of course, a transit connection in some form is an issue, but several spots along GO Lines could be found, if desired, though we're likely talking about despoiling more farmland/nature, though some of this is already approved; see Seaton, north of Pickering which has railway access, though no current passenger service.
More realistically, closer to the action, much of Exhibition Place is parking, and non-heritage structures.
*********
Now look what you've done! Here I am trying to mostly dismiss the idea and you've gone and made me try to see how to make it work, stop that! LOL