Newborn babies have very immature immune systems as they haven't been exposed to much yet. It's like visiting anyone else who is immuno-suppressed. The flu jab is only ever a best guess- manufacturers have to predict which mix of wild strains are currently prevalent and play the odds a bit- but for the sake of £12 or whatever it costs over in the US, get the shot. It won't do you any harm and parents of newborns have enough to worry about with the lack of sleep and general shock of changing circumstances. My wife's a GP and put it in far better terms in between our general confusion, stress, shock, pain (on her part) and lack of sleep. On top of that, they've probably been given that advice by various midwives, health visitors etc too.
The odds are it probably won't make a difference, but if it does, you've put them through a lot of worry at the point in their life where they are least prepared for the extra load, for the sake of a minor inconvenience to a healthy adult. The anti-vaccine crowd seems insane over there compared to here.
In the first weeks after my daughter was born I had so much shit to learn and do that all my friends were practically begging me to say what would help. I said hold off on visiting for a bit if you aren't sure what you are vaccinated against, and preferably find out, and be prepared to chip in with chores if you are coming, as we didn't need anything else to worry about and hospitality was going to be mildly hampered for a few months while we got back into a routine. I'd do the same for my mate's families too.