Makes sense if we accept that racism isn't limited to cartoon villains like Farage, Wilders and Trump. Even they are great examples, three politicians built on anti-immigration stance and all three married to foreigners that shouldn't stand a chance under their own policy. If you think you aren't racist, maybe you don't notice it as quick.
Its also degrees of differentiation on what one counts as 'racist', and most often what you'll find is people presume that because they don't do or say X, which is much more blatantly racist than Y, they're not a racist, but Y is still kind of a racist thing.
Roma are a really weak spot for a lot of Europeans I've noticed. They just have no advocates so all you ever hear is negativity. The only other similar group I can think of who face similar universal disregard are sex workers.
History and local level encounters is what plays into this. Because the Roma were and are commonly - but obviously not exclusively, especially in the present - a nomadic people, they often ran into conflict with an otherwise seated population, especially as the caravans for which they are (in)famous require a great deal of space that, quite often, was not open for them to leave said caravans on. Thus they were often seen as disruptive or invasive, while also being strange and mysterious, sometimes dangerous. They were often viewed as thieves, murderers, rapists - just generally quite criminal - owing to the presumption that their limited means made them desperate or opportunistic, while the camps were so isolated from wider society that you could never know what they were 'planning' within. Add in being 'dirty' because of whatever waste they built up and then left because they weren't exactly in a position to have it disposed of properly. This overall perception of them still heavily lingers in the cultural memory, somewhat softened into simply being squatters who don't pay taxes, but for many and in many places, the Roma quite generally just mean trouble, whether or not they actually are.
An equivalent, often correlated group (as in, also slotted under the vague umbrella of 'gypsies') in the British Isles are called '(Irish) Travellers', for pretty basic reasons, and they often get a similar degree of flak.