Yep. Those two are unique in that they essentially have nothing around them - no connecting buses, very little employment, very little housing. This should change in the near future with Bessarion as the western parts of the Concord site get developed, but that's going to be another 10+ years. Bessarion's ridership is already growing significantly. It increased by 50% from 2012 to 2015.
From the 2015 numbers you can see the obviously low-use stations... Summerhill, Glencairn, Rosedale, Old Mill, Greenwood, Chester, Castle Frank, Bayview, Leslie... what they have in common is that they don't pick up passengers from any major routes, and they don't have much employment nearby (Leslie being a notable exception to this). The stations with high walk-up ridership despite no major bus connections are the ones in employment-heavy areas - North York Centre, Yorkdale, Sherbourne, Wellesley, and Bay.