Forgive me if I am ignorant of comments that have already been made, but doesn't the very peculiar set-up of the Pan-Am facilities actually hinder, not help, an Olympic bid?
Before Queen's Park decided to just admit reality and sell the Pan-Ams as 'Toronto 2015,' it was, of course, a "Golden Horseshoe" bid. This resulted, as we know, in ridiculously dispersed facilities to spread a bit of Pan-Am pixie dust to as many ridings--sorry, I mean municipalities!--as practicable.
Do we really intend to hold swimming events, one of the two marquee Olympic sports, in a venue at UT-Scarborough that won't be accessible by rapid transit even in 2024? Or to have a velodrome in a field in Milton?
It seems that an Olympic bid with a reasonable chance of success would have to propose new facilities in a central location, IE the Port Lands, per the IOC's well-documented preference for having as many events as possible co-located in an Olympic 'park.' That means duplicating specialized sporting venues which already face non-trivial 'legacy' questions.
There are always some events held at a distance from the central venues in any Olympics, typically those that require large outdoor courses (ie rowing, mountain biking, etc). But I find it hard to see how Toronto could succeed without significantly centralizing some of the core sports around the proposed Olympic stadium. That means additional cost to duplicate facilities, and undermines a significant part of the supposed advantage of having hosted the Pan-Ams first.
It will be interesting to see how Tory proposes to thread this needle, since he doesn seem quite bullish on on Olympic bid.
An aside: as an ex-rower with a keen interest in the sport, I would LOVE to see the 2008 plan for a proper 2K course in the Shipping Channel revived, making one of the few summer sports in which Canada is consistently competitve a centrepiece of thr Olympic park. But considering St-Catherine's has a totally world-class course already it seems unlikely this would happen, and rowing will be relegated to a distant site -- as in London, Beijing, and Athens.