The whole thing, if I remember, generated from a line in Let's Kill Hitler when Rory (after Mels regenerates) complains of a "banging in his head". Then there was, what people believed to be, "overly aggressive behavior for Rory".
I think that was it.
Also River being a time lord too. The being conceived on the TARDIS always felt like a weak explanation to me.