Just in case it helps anyone out that's having problems, this is the process I used for updating my iphone 4. The download took 15-20 minutes, the install a little longer though that was mostly the restoring from backup stuff. I used a Mac.
* Made sure iTunes was updated before ever plugging in the phone, then restarted the computer, restarted iTunes.
* Cleared out any notification badges on any apps on my phone, cleared the apps that were suspended, rebooted phone with homebutton+powerbutton.
* Plugged in phone, Checked for Update
* Did Download Only for update (not download and install). Update download took about 20 minutes.
* Decided to do a regular standard sync before anything else. Clicked on Sync. iTunes told me there were some things purchased on the phone that I needed to sync over first, so I went ahead and did that. So a full regular sync, created backup. Then I did it again.

* After it was done, went ahead and clicked on 'Update Phone'. It was already downloaded and ready, so it went fairly quickly.
* After the phone did it's install and automatic reboot, once it got back into iTunes iTunes informed me it was going to restore from my backup. This by far took the longest. In fact, the phone actually went to the normal homescreen while it was 'restoring apps' in iTunes.
Some things to consider:
As mentioned earlier, if you've actually been backing up the phone regularly, including the data tab (ie transferring contacts to Outlook or Windows Contacts or the Address Book, etc.) and have your non iTunes music backed up somewhere, don't be afraid to just do a clean restore and just sync later on to get your data back. Especially since redownloading apps and music that you bought in the past is fairly straightforward now.
Probably most of this everyone already knows, but hey if it helps anyone out that's cool too.