To me the ultimate question is whether the baker will make them a regular birthday cake, or whatever. If he does, then he is not discriminating against them as people, just choosing not to participate in a specific thing he opposes.
If he refuses to bake any cake for people because they are gay, that is illegal discrimination. For the same reason a Jewish bakery should be able to turn away a swastika cake for the local alt-right party.
If you enforce one instance and not the other, then the state is choosing between viewpoints, which is impermissible.
Neo-Nazis are not a protected class under the 14th amendment or CRA of 1964. Based on precedent, LGBT people are.