A few thoughts for you, rather than focusing on other areas you could see (though I'll mention 1 or 2 at the end), some suggestions on what to do in areas where you are already looking to go.
Yorkville: - Make sure and stop by Yorkville Park, located on Cumberland street, between Bellair and Avenue Rd. Its small, but quite interesting, a great place to people watch or sip a soda in the shade (has tables, sitting rocks and giant rock from Northern Ontario you can climb up and sit on)
- If you enjoy fine food, but don't want to blow the budget, you might try D'Lish cupcakes in that area, wonderful product, very inexpensive (takeout), I'm going to guess there is egg in them, so not vegan though.....
- If you like looking at expensive baubles, pop through Holt Renfew, its sorta of Canada Answer to Sak's 5th...., right on Bloor Street, just east of Bay, three levels of mostly over priced, but nice goods.
- Want a nice place to grab something vegetarian in a quaint setting, try 7 West, its on Charles Street, just west of Yonge, its in a old Victorian House, its also open 24 hours, if you need something early or late.
- Fancy food (grocery), pop into Puseteri's, its a mini-supermarket, on Bay @ Yorkville, (only about 5,000sq feet) but has some very interesting stuff.
- Book lover? pop by the Toronto Reference Library, on Yonge @ Yorkville, nothing much to look at on the outside, but a palace of good reading material inside.
********
Yonge Street
To be honest, south of Yorkville, but north of the Dundas area, its mostly humdrum, though there are a few 'jewel' stores.
But I would tend towards getting down the street to the Dundas area, where more interesting retail can be found.
Consider popping into the Hudson's Bay store (corner of Queen and Yonge and connected to the Eaton Centre), one of the largest department stores anywhere, 9 floors to keep you amused.
For buying books, one the uglier but better stores you'll ever find is the 'World's Biggest Bookstore' on Edward, looks very blah, but it does have a very large number of titles (though not the largest in the world by a fair margin)
If you enjoy heritage buildings, consider Old City Hall, which is just beside the Eaton Centre on Queen Street west...........
or for modern architecture, consider new (current City Hall) just across Bay street, with Toronto's main public square in front of it.
****
In the Financial District, don't miss walking through Brookfield Place, at the ground level, it has a corridor connecting Bay Street to Yonge Street with a wonderful skylight feature, one of the better indoor public spaces, and quick glance is a but a minute's time. Its located in the block bounded by Bay/Wellington/Yonge and Front.
From the outside, TD Centre (King and Bay, and Scotia Plaza (Adelaide between Yonge and Bay) are among the more interesting bank towers.
****
Distillery District
- Soma Chocolate is very nice
- Balzac's coffee as well
But mainly a walking/event area
****
St. Lawrence Market
- A must see, Saturday morning, hitting both the Farmer's market (non-descript building on the north side of Front street @ Jarvis) and the main market on the south-west corner.
- While there, stroll down Market Street (west side of Market), then along the Esplanade.
- I would use the Esplanade to walk between the Distillery District and St. Lawrence, its the nicest route, while Front just west of Jarvis is also nice.
- One of the best photo corners in the City (views) is Church and Front (1 block from St. Lawrence), interesting angled heritage building you may have seen in the opening sequence of the show Shark Tank, and great views of the CN Tower and Financial district beyond.
****
Your schedule is already quite full.
But aside from endorsing the various suggestions made Kensington Market, Chinatown and the Islands.......
I will put a one or two more out there, depending what interests you, old buildings?? Stroll through the University of Toronto campus, focus on King's College Road/Circle, and then the path known as Philospher's walk, running from Hoskin to Bloor (behind the Royal Ontario Museum or ROM.
The Islands are nice, great City views (good beaches if the weather is warm enough), and some lovely nature as well, but.....it is a fair time commitment between the ferry schedule and a good walk.
Enjoy 'ethnic' variety? consider Koreatown, Little Italy is also nice, though not all that Italian anymore, there are many other 'cultural' shopping areas, however most are well outside the downtown core and would entail sizable travel times.
***
On vegetarian, but not necessarily vegan food choices, consider this list from a local blog (Blog TO) of Toronto's best Vegetarian places.
Take this and any list from them with a grain of salt, but there are often some legit. gems.
http://www.blogto.com/toronto/the_best_vegetarian_restaurants_in_toronto/
Hope that's helpful.