I walked the area on Saturday afternoon.
South of Sheppard Avenue, via the Hallmark Centre you can get into the Rexall without going outside, but there's no internal connection to the Whole Foods.
North of Sheppard Avenue (and not connected underground to the Hallmark Centre without using a Metropass), the connection from the Sheppard Centre to the Madison Centre and Federal Building is one of the ugliest pedestrian passageways in the city. The connection point at the Sheppard Centre is, like at 110 Bloor West, outdoors, but covered from above. With a long-closed restaurant by the beginning of the tunnel, it looks like the beaten-up bits of Buffalo or Detroit down there.
The Madison Centre is as far north as you can get before you're outside again. Ultima and the buildings to the west of it have not joined the party.
The North York Centre and Empress Walk have direct connections to North York Centre station of course, and via another covered-but-outside connection, you can reach North York "City" Hall. There's a direct indoor connection from it to the Toronto District School Board offices.
42