This is an interesting topic I can't come to a satisfactory conclusion on.
Firstly, OP is incorrect that we don't punish or find morally problematic people who damage the genetics of their future children. This paper makes a case against the practice (
http://reproductiverights.org/sites/default/files/documents/pub_bp_punishingwomen.pdf), but concedes that the phenomenon in fact exists. Many jurisdictions have passed laws criminalizing women who drink or do drugs during pregnancy, andthe trend seems to be toward passing more of these kinds of laws.
Intuitively, it makes sense to me that a drug habit during pregnancy that leads to brain damage is morally problematic in a similar way to someone who breaks a beer bottle over someone's head and causes brain damage, though of lesser culpability because less malice is involved. So I'm pretty comfortable in asserting this much: damaging the genetics of your offspring is functionally equivalent to introducing the harm in any other way. The mechanism of harm shouldn't matter that much; the harm that is inflicted should be what's important.
This is not, however, entirely satisfactory. In particular, the genetic argument has given in the past has given legitimacy for oppressive behavior on the part of the majority. A first-blush reading of the moral principles I've laid out here would similarly prohibit allowing those with inheritable disabilities, or HIV positive people, from having kids. I'm not entirely willing to rule this out as an end result of my moral principles, but for now it's important to note that the application of the principle to these populations introduces substantial civil rights questions. I'll toss the incestuous population in here, too, whose oppression may ultimately be justified but nevertheless in the main be motivated by animus against an unpopular sexual minority.
The point about other risks for birth defects is well made, but allow me to make an analogy. Getting behind the wheel of the car endangers other people. Doing so while tired endangers them even more. Doing so while drunk endangers them even further. So at what point do we draw the line between an acceptable risk, and at what point is such a risk unconscionable? I'm not sure, but I do know that there is a line somewhere. That we do not imprison the tired driver does not make us hypocrites for imprisoning the drunk driver. An increased probability of harm can be considered unwise, while still below the threshold of moral condemnation, while an even greater probability of harm can rise to that same threshold.
Here's what my research found. I haven't vetted these sources, and I'm not entirely certain these studies use "birth defect" in exactly the same way, so caveat lector. That said,
20-36% of children born from parent-child or brother-sister incest will die or have major birth defects,
6 to 8 percent of children born to mothers aged 40 and above will suffer a similar fate,
4% of children born from cousins will have birth defects, and
2% of babies born in the general population as a whole will have birth defects. Based on the reasoning above, the taboo against direct relation incest would seem to be justified, rising far above the probability of harm that a 40-year old mother does, but the taboo against cousin incest seems largely unjustified. This combined with the wealth of societies that have no problem with cousin leads me to conclude that its current prohibition is largely the product of a closed-minded populace expressing animus against an unpopular sexual minority.
At some point I want to tackle the issue of bodily autonomy and our reluctance to impose eugenic limits on reproductive freedom in regards to this issue, but I've rambled on enough for now.