Crabs in the barrel, man. My classmate's brother said that he faced the same thing when he was about to be released from prison. Word gets around, and people want to make your last days miserable. Hell, some people get killed days before release.
I mean I'll be honest, it's hard for me too. My mom and brothers were homeless for a while. Then we lived in the livingroom of our grandma's house, us 3 having to share the same couch hideaway bed while our mom slept on the other couch. So it's like leaving the tribe in a way, after all we've been through earlier in life.
I get that they feel hurt, and it isn't easy for me, but if I want to do the stuff I want to do (pay off their house, make sure they don't have to go into a home), I have to leave completely, have my own life for a while ENTIRELY outside of social web of this city and start a new one elsewhere.