I did recently and it was a jarring experience. So sad to think what a glorious avenue it could be with minimal investment, yet the city couldn't care less.
The city could spend a small fortune on the public avenue and it wouldn't do much. Too many buildings turn their back on Jarvis. There's some odd townhouse infill as well. It has a forgotten luxurious past. That's it.
It could be fixed up to an extent from Bloor down to Gerrard by infilling with midrises to consolidate its streetwall, improving the street realm and restoring the old mansions along the lines of the Selby and James Cooper Mansion.
But you're definitely right that there are some nasty townhouse developments along Jarvis.