It's possible to be very invested in a hobby while not liking the stigma attached to it. And yes, there is one whether I or anyone else wants there to be. It comes down to the individual person whether or not they are bothered by this.
Even beyond that, I don't like the label of "gamer" for a couple reasons. First, people use it to be assholes, like "You're not a real gamer!" which bugs me to no end, and the further I can distance myself from those people the better. Second, no other media has a self-attached label to it like "gamer". "Film buff" or "movie snob" aren't really comparable because it's not a single word, I can be a "snob" about anything. I can be a food snob, television snob, book snob, etc. But games have the only single word label, which bugs me, and only reinforces both the negative stigma and the assholes who use it like an exclusive club.
Basically, "gamer" to me is the negative parts of the culture that I don't like. Not being a part of that, but still being really into games, is important.