I've just read the Tyrion chapter in SoS where he talks with Tywin who denied him Casterly Rock, and where Tywin comments that he could never prove that Tyrion is not his son, so he is forced to treat him as one of his family. Now that by itself may not mean much, just an indication of the dislike that Tywin has for his dwarf son. But I believe this is the second time that Tywin makes the same comment, and it seems strange that Tywin would keep making the same otherwise petty comment - unless he seriously felt that Tyrion could possibly not be his son.
Tywin's last words to Tyrion are also "You are not my son". Admitedly he says the same thing to Jaime, which in fact makes me wonder if this is not a deliberate red herring.
But we also know that Tywin was 20 years the hand of the king. We know that Aerys was envious of him. We know that Aerys wronged him many times, and that this led to Tywin abandoning his post and position, apparently the only man Aerys feared so much as to not try and punish him for it. We are led to believe that what pushed Tywin to leave Aerys' service was Jaime entering the kingsguard, but again we have that from third-hand.
I think the theory that Aerys at some point forced himself on Joanna is a very possible one. Tywin otherwise seems to be at least logical, he knows that death at childbirth is not a conscious fault of the child, to hold a grudge against Tyrion for it. But if he had reason to suspect the child was not his, he would have reason to have such hate for him. It would also explain his own hatred for Targaryens, leading him to order the murder of Rhaegar's children, and letting King's Landing be sacked (which would to him be a city that stayed loyal to Aerys).
At the same time in Feast, Aemon tells Sam that both he and his brothers dreamed of dragons. We know that Daenerys dreamed of dragons before she had her eggs hatched. And if I remember correctly, in book 1 Tyrion also tells Jon that he used to dream of dragons - or at least that he was fascinated by them. If I recall correctly one of his mismatched eyes is also the Targaryen color.
It also seems to have a nice symmetry to it with Tyrion now going to find Dany, and a nice symmetry to the story, with all 3 main characters (Dany, Tyrion and Jon), being the last descendants of a once mighty but fallen dynasty, all 3 of which had to suffer being outcasts in some way in their childhoods.