Losing Kennedy wouldn't hurt that much aside from gay rights, tbh. Guy already has a dim view of more or less everything else, and it was Roberts who saved Obamacare.
And you're not going to reverse Obergefell at this point, basically impossible to put that back in the box, logistically (how would states annul all of those marriages? Imagine a wave of involuntary divorces happening all at once, nightmare for an overpacked court system).
It's Ginsburg dying that would screw us for the first half of the 21st century and basically require court-packing to undo.
Edit: the other alternative is also having congress basically wipe the slate clean on the lower court system and rebuild. Fire everyone in the circuits in the name of "modernizing" the circuit system, and stuff the whole thing with liberal justices.