You're implying that modern-day polarization is due to factors that are affecting both parties equally, and so both parties are at fault. That's false.
The parties themselves aren't necessarily at fault, it's more of how the game is played (primary elections). There are less moderates in both parties (in Congress) today than there were in the past. There's a few reasons for this and I think the nature of primary elections is partly (not solely!) responsible.
I even mentioned that Democrats have had more success in electing moderates than Republicans in recent years, but that in itself is due to the nature of primary elections (where far-right politicians have beaten those more moderate). Republicans have had
some success - Scott Brown was a moderate. I know that a lot of liberals think he's far-right, but he really wasn't. He wasn't a liberal, but he was moderate. Mark Kirk is another pretty good example of a moderate Republican. And go ask Joe Lieberman how it was to be primaried from the left (regardless of whatever opinion you hold of Lieberman).
Barbara Boxer's 1992 Senate election perhaps best demonstrates why the primary system is partly at fault for increased polarization (and more importantly, why it isn't a new thing). Boxer was (and is still today) one of the most liberal members of the House (and now Senate) and she probably would have been beaten (or at least closer) by a moderate Republican (Tom Campbell), but was saved when Bruce Herschensohn (a hard right winger) beat him out in the primary. And that was in 92. This has been going on a quite a while.
If one party consistently pandered to the base as you describe, and one party consistently attempted to force moderates through the nomination process, the party that pushed moderates would keep winning elections and marginalize the other party until it changed and started controlling the nomination process more aggressively again. This isn't even theory, this is what's happening to the Republican Party right now! Sure, it's a little more complicated at the district level, but fundamentally the median voter theorem still holds sway. If a party's national positions don't align with the majority positions in swing districts, it's going to mitigate against its candidates winning those swing districts, because they have to juggle their allegiances. Therefore it's just good electoral sense to make sure your national positions are majority positions.
The median voter theorem was much more pronounced in the middle of the twentieth century. I would argue that the strategy today (and for the past couple decades) has been very focused on the base, not the median voter. A candidate who moves to the middle might lose more votes than he or she gains. As I've mentioned before, politicians are worried about two constituencies - their primary election constituency (ie the base) and the general election constituency (which is
not the entire electorate, but the number of votes they need to be elected whether 50%+1 or otherwise). Neither candidate can afford to lose the base. That's why they aren't moving to the center.
Now, the base turnout strategy on a national level - as you mentioned - is beginning to dwindle (at least for the Republican Party). But I find that it still holds true for most house elections and some Senate elections. National elections are very different than those two (and Senate elections different than House ones, although radicals still win nominations there).