We had a really awful combination, today, of unprecedented traffic (expected, of course) and the physical failure of one of our database servers at the worst possible time. We're on 50% power with the database servers right now. Keep in mind that each one of them is really strong, but we hit an i/o hard wall during the conference and shortly after because of that. Entire internet was spamming F5.
I'm having gromph redeploy our database tomorrow to even more powerful servers, essentially the most ridiculous array possible, no expense spared to round out any possible weakness, to try to prepare us better for what's to come.
Major, major disappointment that we were down for almost the entire conference. It would be great if we could use an elastic solution to dynamically address load with no upper limit, like Amazon EC2, but apparently that's not viable for our setup.