Focus on the friends!
Look, I can't sit here and say "I've been there" because I"m open on GAF, with my wife, and with a few friends that I trust. It sucks when people talk shit, and it sucks more when what they say kneecaps your good time. All I can say is that I've been there, knowing that people are talking about you behind your back, and hearing it. Happened to me in high school all the time.
But there's more good in the friends than bad in the fuckwits. Those fuckwits have the power that you give them, and they don't deserve it. Much as I advocate against smoking, go have your smoke if that's the least of the drugs you're talking about, and don't give them a thing that they don't earn.