you'd think so. Try ordering food or a taxi to that address and see what they say to you. It's not even that they won't go to the island. It's that the address doesn't compute for most places. So instead we have to say 1 Bathurst St which is pretty much the only address anyone would recognize. Taxis are hit or miss really.
You'd also be surprised how many people (passengers, taxi drivers, etc): 1) Don't know the island airport exists, 2) don't know there's a ferry that takes you there, 3) don't know where they can catch the ferry. So not everyone knows where it is.