As a York student myself, prepare to mutter that statement frequently.
Not really too much to add. The transit trip is pretty straightforward and while it reading it here might look like it takes a while, it's really a snap. I live east of downtown by Donlands station and door-to-door it takes me between 60 and 70 minutes, taking the Bloor-Danforth line to St. George, the University line up to Downview and the 196. So, 40-45 minutes sounds about right if coming from the Annex. The biggest hassle is trying to
leave York at certain times of the day, particularly around 17:30. There's a huge bank of classes that lets out at that time and the line-ups for the 196 can sometimes be insanely long. To add insult to injury, the buses are often bunched so you'll stand in a 150 m long line with no buses for 5-10 minutes, and then four of them will show up at once. So, if possible, try to avoid leaving campus at that time.
Another suggestion which, depending on whether you end up further west, like around Bathurst or Christie stations, is to consider taking the 41 Keele bus. It usually won't save you any time, but it can often be less annoyingly crowded than the 196. However, again, avoid taking this bus after about 17:00. Rush hour on Keele can be awful.
There are plenty of options for getting to/from York. Experiment with them and find out which one suits you best.
Oh, and keep in mind that on Sundays the subway doesn't start until 9am. So, should you ever need to be on campus early on a Sunday (I have), definitely do a trip planner first to find out which "night" bus is most convenient for you.