Because homeless people make the street look so much nicer by comparison.
Housing is not an investment and has never been an investment. People who retire aren't going to retire on the back of their investment in housing. They're going to live in their house, which prevents them from benefiting from its appreciation.
When people go to these neighborhood board meeting they're often pretty explicit that they want to keep poor people and people of color out of their communities. As a society I don't really see what we get out of giving them that option, and it's very easy to identify the problems that result from having it.
People do reverse mortgages and the like to squeeze equity from their house, and they will definitely benefit from rising housing prices there. It's not just where they live. It's certainly a bad idea to use it as your sole investment vehicle, but people are stupid about saving money, even people who make more than enough (cue that "even middle class people couldn't come up with.$500" article.)