No-one is saying that immigration was the only reason people voted to leave the EU, but it was the most significant reason, and if people weren't concerned about it, it would have been a very comfortable Remain victory. One of the many history rewrites undertaken by Leavers, politicians, and much of the media, is to imply that immigration wasn't as important as 'taking back control'.
The irony, of course, when most people have no idea what control they want to take back, is that much of the sovereignty argument boils down to xenophobia too.