With College-Front-University-Jarvis as the starting point, how much further to take it, using a main shopping/business/entertainment/institutional area" definition. Going north from there it does include Yorkville and south pretty much to the lake. It also stretches further west the further south you are, maybe as far as Spadina in some points (i.e. King and Queen West but not the Annex).
Areas like the Annex, Kensington Market and Cabbagetown strike me more as defined neighborhoods than just downtown.
But for simplicity's sake, the Bathurst to Don definiton works as a sort of extended or greater downtown with simplified boundaries. Sure you can nitpick over why the Annex and Cabbagetown are included and Trinity Bellwoods are not. But these areas don't really fit in the west end or north Toronto or east end either, so they're sort of central if not quite downtown.
Toronto doesn't really have a well defined downtown like say Chicago or Vancouver with "harder" boundaries. Which is why there is often a very smooth transition from downtown to neighborhood, particularly going west.