F13 UPDATE - Monday, May 29th, 5 pm EST
OK, we made some big strides late last night and into the morning. We found several issues that were causing the database errors, missing XP, infinite saving while customizing characters, and perks not saving or appearing properly. Our number one focus was getting players past the database login error screen. Once we got to around 50K concurrent players, the database would not scale properly. We shut down all servers this morning to migrate the database to a larger cluster to help with scaling. Once we started servers back up, we could support 90K concurrent players. Thats good!
However, we are aware of the following issues that persist.
Login Database Error - Cant sign in
XP not saving or lost XP
Infinite saving when customizing counselor, Jason, equipping kills, etc.
Perks not saving or not appearing properly once saved.
Cant use Quick Play on XB1
Whats going on?
After 10 hours of digging around in the database, we discovered a few things. Right now we are pushing too much data across our database. Your player profile save data is too big, especially if youre a higher level with lots of unlocks/perks/etc. It works just fine when there are only 30-40K players. Once that number grows, the database starts lagging. This lag is what keeps some players out AND causes all the weird XP, saving issues outlined above. In addition, we still need to scale to meet the demand which weve had a phase 2 of the database upgrade planned that we are working on today.
The Quick Play issue with XB1 requires a patch to fix. Meaning we cant fix this live on our end. This requires our partner to help.
OK, how do you fix this?
We have made some good optimizations for retrieving and storing the player profile. We have cut down the calls made to retrieving and saving happen a bunch less. In addition, we have cut down the amount of data that is being saved to the database.
Cool, when do we get these fixes?
Steam users will get a patch late tonight or early tomorrow. PSN users will get this patch tomorrow. XB1 will probably be Wednesday. This isnt because of favoritism, its simply how patches/hot fixes get rolled out. There is no certification process on Steam. Once we have a fix, we click a button, it goes live in 30 mins. PS and XB both have a rapid patch system, but they need to be certified by both PS/XB and this will take time due to today being a US holiday.