Shelters are often unsafe places, but I do agree - many people have chosen this option.
Many people that are homeless are mentally ill and/or struggle with addiction, and violence can be a problem. I wouldn't be totally blithe about it.
However, in de Caire's context I think "difficult-looking" people in that context doesn't primarily mean the homeless or panhandlers (many panhandlers bus in from places they live, and not just in Hamilton do they do that).
This is who "difficult-looking" is for me.
First, those intimidating-loitering type (blocking the sidewalk with your dogs and your buddies). Catcalls, threats and swearing optional, but de rigeur
Two, those who keep pushing drugs or sex (I just love it when my husband tells me that the girl who just got on the bus with us had propositioned him the day before. That's intimidating shit if you keep getting approached or have your spouse being approached in the middle of your most visible public area)
The kids who keep asking if we have weed are just irritating, though I wish the few stores known as hotspots to get weed or "other stuff" would get a thorough investigation and cleanup.
That's who the "difficult-looking" people are for me in the core. Other people like unmarried teen moms (most of whom don't smoke around their kids), the disabled, or the homeless may bother some people on other matters, but they're not intimidating me, getting in my face, or interfering with my enjoyment of public space.
The homeless issue is tough. I can't in good conscience give someone money that I know is probably going to alcohol or drugs, especially when I see the same people at LCBO buying bottles all the time. So I still don't carry change, but I do donate to nonprofits whose programs are actually helpful.
That's me personally... so I'm not unsympathetic.
However, for our city to allow our core to be dominated by those asking for handouts (often much more in the summer when the "seasonal panhandlers" come on out...) that sets a bad precedent and bad image for the rest of the city and especially for first-time visitors.