Hodor was like an puppy, super happy, but terrified of things like thunderstorms. For the most part, of course there were other times when he was scared, or felt remorse after Bran warged him and killed a man. But throughout the majority of his life post accident he appeared to be a happy go lucky guy because he just didn't know better.
Bran on the other hand, post accident has seriously suffered and felt it all because he is fully aware of everything that has happened not only to him, but to everyone, everywhere. Hodor might have carried him, and Meera may have pulled him but Bran is carrying the emotional suffering of how many people, over how many years? And he's bound to a chair or a bed for the rest of his life.
Did Bran really fuck him over? (edit: yes he definitely fucked him over, but still) He went from stable boy, to stable boy, but one that was often smiling without much to worry about. Bran, because of those string of events suffered a fate worse in my opinion.