It's not a crock of shit, though. I think people get confused with two different thought processes: 1) he's small/lightweight therefore he doesn't fit in the physical PL, and 2) he's easily knocked off the ball by strong players.
Kagawa, for example, may in time be a brilliant player for United, but as of last year he was responsible on several occasions for losing possession by getting easily muscled off the ball. There are of course examples of players who are small, but extremely difficult to dispossess, like Modric. Skrtel looks like he should be a tank, but he gets bullied by large strikers.
Small players who can deal with the physicality of the PL:
Rafael, Modric, Suarez, Ramires, (Neil) Taylor
Small players who get bullied easily:
Allen, Kagawa, Oscar, Ramsey