Your just looking at the simple distances though, and not looking at how people move, etc. If your standing on the east side of Yonge/Dundas, you can often walk to Yonge/Victoria faster than the streetcar can move there, even if you've missed your stop. If you are already at Victoria, you really want to fight that crowd to walk back to Yonge? The streetcar often has to stop on those lights anyway.
And King/Simcoe heading west? You don't just walk back to University. You have to cross University as well. This is a VERY well used stop.
Sure, there's some stops that should go. But the secondary stops near subway stations are not the ones. They are a godsend when it's very busy, as you can just get off and walk to the station long before the streetcar gets there. It's those lesser used ones out at the periphery.
Personally, I live near Main/Gerrard, and I'd be happy to lose every second stop between Coxwell and Main, if it meant faster service (and I use one of those regularly!). I haven't noticed too many people with walkers and wheelchairs using the streetcar. And strollers? Not sure why that would need closer stops!