You and I have had very similar life experiences.
It's not usually something that bothers me until I start getting tired of the same thing over and over, and then I just get tired of people in general and I'm more than happy to just wander off and do my own thing. I've never been a lonely person, so the constant feeling of looking in from the outside at something, or looking at something on a much larger (And sometimes unnecessary) scale has rarely been an issue.
But at the same time when you're in that position, you see people do the dumbest bullshit over and over and wonder why nothing works for them, and crazy stuff like that which baffles the mind, so on the one hand I don't care for people because of what I see over and over, but with the other I sometimes want to pick them up and point them in the right direction if I can.