Wait, so you didn't like the fact that people weren't as attracted to you when you were overweight. I take it, then, that you've never made a comment on someone's attractiveness before - based on any physical characteristic - and that you literally see no difference in attractiveness based on a person's appearance.
Congratulations, you're a rarity.
Now I'm not saying you're not a great guy, nor that larger people can't be attractive at all, I'm just saying that based on your age and demographic, I'm thinking that it's hardly surprising, nor entirely unfair, that people weren't as attracted to you when you were bigger. I mean, for the most part, most people in western society don't find larger people attractive.
It's like being offended if you offered someone a plate of fried grasshoppers and they said "hell no!". It's a cultural/societal norm to have a notion of what is attractive. If you weren't it, then don't blame other people.
And yes, I've lived in that not-so-attractive fringe for years my self, so I'm not entirely unaware of your position.