I think you can come to that conclusion without the Tower of Joy explanation. It's not exactly rocket science that when a man and a woman mysteriously "vanish" together, some hanky panky is probably going to occur.
Possibly... I mean, what made the R+L=J so conspicuous in the books is in the fact that Lyanna made Eddard
promise her something on her deathbed, where a bed of blood was also a metaphor for a birth bed. So in the very first book, it is strongly implied that Lyanna had a child, obviously with Rhaegar, who, as Robert so graciously put it, 'raped her'. No, I don't believe it was rape, but it couldn't go over anyones head that Lyanna and Rhaegar had sex, whereas in the show I don't think that was even mentioned down in the crypts when Ned and Robert stand before the statue of Lyanna. ( At least that was one part the show wasn't very obvious on, if at all ). Personally, I don't think anyone of the 'truly' unsullied could have put any significance to Lyanna other than Robert loving her and Cersei being bitter about it. Just too many leaps to be made that I think I would actually have to call tinfoil on it, if all I had was the information from the show.
The Kingsguard at the Tower of Joy has another significance though, which would be that Jon was no mere bastard and Lyanna wasn't just some paramour out in the middle of the Dornish Marches, so that should get mentioned soon too, somehow.
Still, now that 'Rhaegar and Lyanna' is again out there, any fool can type 'Rhaegar + Lyanna' into google, and lo, Jon Snow theories abound! And fan art.
This pairing is like Martin's own Lúthien and Beren, someone once said, and I am inclined to agree on that. More tragic though.