You are definitely being too sensitive. They are not harmful. Majority of the crime in the area is drug related (dealer on user, user on user) or petty crime (mainly car break-ins)
If you want to live in a city that rounds up their homeless and puts them into a segmented ghetto, you'll want to look towards some of the US cities. Here they are concentrated, but it's not a segmented situation. All walks of socio-economic status co-exist generally peacefully here and it's one of the things that makes us better.
Stop being "poor people are gross" ;p
Only gross poor people are from Surrey
I mean a bit of joking but one of the big issues with Surrey is due to the socio-economic situation, sub-urban culture, poor police funding, and plus drug culture all mixing. It leads to a lot of violent crimes. I feel comfortable walking around at 1am in DTES as all that may happen is someone asking aggressively for money. While walking around areas in Surrey at night I'm worried about running into the wrong people and getting stabbed. While here the violent crime tends to be contained among drug users and/or dealers.
Most of the homeless in DTES generally are just people with mental conditions that cannot function in society normally by keeping a job or need drugs to deal with their mental health issues. If riverview was open (it's being re-opened) and funded properly (it's not being funded properly), then you would see a lot less. You should have some sympathy for them.
It's not a case like a punk kid in Surrey wanting to stab you for your phone. Your personal safety isn't at risk.