From the manga (it's not a spoiler, but just in case):
Goku reveals he didn't know about Gohan or Goten's births. Vegeta: "You might not be a bad guy, but as a father, you're the worst."
This not only sounds plausible, but I also wouldn't be surprised if Toriyama said it was his idea.