Yes it's true that the United Kingdom was the 16th country in the world to adopt same-sex marriage under a joint left-right coalition government because it was pushed by the left part of the left-right coalition in a vote that the entire Conservative backbench opposed and the UKIP strongly opposed. I guess my point is that when it comes to the Conservatives, even if you put lipstick on the pig, it's still a pig.
I agree that it is unlikely that a democratic government in a country with same-sex marriage would move to remove same-sex marriage, but as the very wise people arguing for Brexit have correctly pointed out over the last 24 hours, it actually doesn't matter if someone's fears are real or not, we should listen to them anyway and treat them like they are.
P.S. Worth noting the Brexit campaign did campaign on "The EU is bad for gays", including Boris specifically.