I would say roughly the area bounded by Church, Adelaide, Parliament and the tracks.
We moved here somewhat recently from the Annex and I don't think there's much difference in terms of security. We have a little one, so with Covid are never out in the evenings or at night, but we are out a lot during the day and have never had any issues. Obviously there's a fair amount of poverty in the area, but virtually everyone has been perfectly polite to us at all times.
Just today I was asked persistently for cash by someone in St. James. After a couple of "I'm sorry"s, he told me my kid is very cute, wished me a nice day and moved along. That's pretty much been most of my experience with the neighbourhood. I know if you read facebook groups there are all kinds of reports of harassment and dirty needles littering the area, but we are in the parks and playgrounds twice a day every day and have never seen anything like that. I don't want to diminish anyone's individual experience, but overall the issue seems overblown to me.
There was a guy hanging around the Tim's at Front and Sherbourne today with a hockey stick, which alarmed my wife a bit, but I'm pretty he had no intention of doing anything with the hockey stick other than wandering around and maybe daydreaming about winning a Stanley Cup one day.