Steam security issue revealed personal info to other users on XMas Day (fixed)

daviyoung said:
prior preparation prevents piss-poor performance

it's not beyond their scope to plan and test for this and make sure that infrastructure can support worst-case scenarios

making a config change at this date/time and with this much impact was amateur hour
Click to expand...

Well, the way it was handled was poor, no doubt, considering the result. But that you make changes when the load hits, not so sure that's amateurish, but those who have worked with similiar systems might be able to fill in with more details. My own practical experience are with systems with much smaller usersbases.
 
CecilRousso said:
Well, the way it was handled was poor, no doubt, considering the result. But that you make changes when the load hits, not so sure that's amateurish, but those who have worked with similiar systems might be able to fill in with more details. My own practical experience are with systems with much smaller usersbases.
Click to expand...

The fact is, during each sale is pretty much the same. Valve knows there will be high traffic volume. This sale was the worse, because I needed to wait something like two hours after the beginning of the sale to start exploring Steam decently. I know servers have a cost, and managing new one when there's no need is a waste of money. Still, they can rent more servers under sales in order to offer a better service. But they don't do that because, as it's clear, they don't want take out money, the same reason their support sucks and they are automatizing everything including refunds and games removal.
Remember, they offered refunds because they were forced to do so, not because they care about customers or similar.
 
Kathian said:
Going to be using Steam much less after that response. Which won't be hard considering the prices they charge outside of sales.

There are so many warning signs around how Valve are run. They need more dedicated staff regardless of culture. This should not just happen.
Click to expand...

There are a lot of things that shouldn't happen, but the reality is that most services have major security leaks. Neogaf had a major leak last week that required a lot of people to change their password.

In the last couple of years alone; http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/




Nobody can rely on any site to keep your information safe from attacks. It's not how it should be, but it's how it is.
 
Vigilant Walrus said:
There are a lot of things that shouldn't happen, but the reality is that most services have major security leaks. Neogaf had a major leak last week that required a lot of people to change their password.

In the last couple of years alone; http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/




Nobody can rely on any site to keep your information safe from attacks. It's not how it should be, but it's how it is.
Click to expand...

This was not an attack. And its more the response. They caused it and reacted both slowly and flippantly.
 
fantomena said:
Where's Valves response?
Click to expand...
http://kotaku.com/steam-goes-nuts-offers-access-to-other-peoples-account-1749718979
Steam is back up and running without any known issues. As a result of a configuration change earlier today, a caching issue allowed some users to randomly see pages generated for other users for a period of less than an hour. This issue has since been resolved. We believe no unauthorized actions were allowed on accounts beyond the viewing of cached page information and no additional action is required by users.
Click to expand...

That's all we got
 
I always thought Valve would be like Amazon and have a fairly good security history. After all the Sony hacks though I stopped storing my debit card and address anywhere. Feeling pretty justified about that now. 20 seconds inconvenience every time I check out vs. having some hackers have my personal info.
 
There is T-shirt potential in an "I survived the 2015 Steam apocalypse by doing absolutely nothing".

While people really need to stop going straight to full on panic (and in this case make everything worse for themselves), (once more) Valve seriously need to work on their customer support and communicate what is going on when the shit goes down.
 
Vigilant Walrus said:
There are a lot of things that shouldn't happen, but the reality is that most services have major security leaks. Neogaf had a major leak last week that required a lot of people to change their password.

In the last couple of years alone; http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/




Nobody can rely on any site to keep your information safe from attacks. It's not how it should be, but it's how it is.
Click to expand...

Shit can happen, expecially if hackers are involved. But Neogaf doesn't store your CC or private info anyway, it's a free service so no one should argue if something bad happens, expecially if it's no one's fault.
Otherwise you put money in Steam, you give them trust and with trust comes your personal infos.
I could understand an attack by hackers, I can't understand you being incompetent for the second time, leaking a lot of potential info out in the wild.
What is really shitty is their statement: they didn't apologize, they took this like nothing. This is infuriating and disrispectful toward their customers, plain and simple.
Sony was hacked, but as you said, shit happens. But they took good measuers, they offered a one year plan in order to protect your personal infos. They took their time, but they forced a mandatory password change and recovered the service only when they was sure all was fixed and running as planned. It took a good month, but better then having your infos stolen again in no time.

Valve otherwise acted like nothing happened, they even didn't stop the service, of course pushed by the fact they would lose money by doing so, since sale is the period in which they do the majority of their incomings.
And after this, they released such a statement.
 
Ludens said:
Shit can happen, expecially if hackers are involved. But Neogaf doesn't store your CC or private info anyway, it's a free service so no one should argue if something bad happens, expecially if it's no one's fault.
Click to expand...
So because Facebook is free we shouldn't care when they leak your private communications? Ridiculous argument.
 
I'll speak on behalf of SteamDB here. We always tweet and post information on our blog only after verifying it ourselves, we didn't base our information on speculation, but rather on our own research. It was pretty clear that it was a caching issue, as it was caching the first non-cache hit on any url (you could easily test this by adding ?something=random to the url, and then loading it from another browser/machine while not being logged in).

I believe this was caused by a misconfiguration on Akamai (Valve's CDN), which caused this to happen. You can use Akamai's debug headers to get some interesting information, and during the issue it was showing X-Check-Cacheable: YES at all times, which is not good. There are no indications that anything could be done on your behalf (caching is read-only).

Yes, we know we know we shouldn't be speaking for Valve, but we find it more important to keep users aware of the issue. Our track record is pretty damn good in that sense, and if the response Gamespot/Kotaku received from Valve is indeed correct, we pretty much nailed our caching theory.

Before publishing the blog post, we had multiple people that work closely on Steam (besides us) proof read it too.
 
PaulLFC said:
In what situation would you not start the time calculation at the first reported occurrence of the issue?
Click to expand...

If I make a statement about an issue I caused I usually start the timing when the information reaches me to the point where the fix has been shipped. When splitting hairs like 30 minutes it's always important to remember that rolling out things takes time and thats how it is. Would the statement made you feel safer if it laid out things by the minute? Maybe.

xPaw said:
I'll speak on behalf of SteamDB here. We always tweet and post information on our blog only after verifying it ourselves, we didn't base our information on speculation, but rather on our own research. It was pretty clear that it was a caching issue, as it was caching the first non-cache hit on any url (you could easily test this by adding ?something=random to the url, and then loading it from another browser/machine while not being logged in).

I believe this was caused by a misconfiguration on Akamai (Valve's CDN), which caused this to happen. You can use Akamai's debug headers to get some interesting information, and during the issue it was showing X-Check-Cacheable: YES at all times, which is not good. There are no indications that anything could be done on your behalf (caching is read-only).

Yes, we know we know we shouldn't be speaking for Valve, but we find it more important to keep users aware of the issue. Our track record is pretty damn good in that sense, and if the response Gamespot/Kotaku received from Valve is indeed correct, we pretty much nailed our caching theory.

Before publishing the blog post, we had multiple people that work closely on Steam (besides us) proof read it too.
Click to expand...

I agree that users should be aware, and the statement should have been posted somewhere else than on Gamespot or Kotaku too. However, not sure what the follow up should be. Technical details confuse the shit out of people (see: This topic), there's no additional steps that you can take at this point as the damage is done and the issue is fixed. The statement laid all of this out already.
 
Visualante2 said:
I always thought Valve would be like Amazon and have a fairly good security history. After all the Sony hacks though I stopped storing my debit card and address anywhere. Feeling pretty justified about that now. 20 seconds inconvenience every time I check out vs. having some hackers have my personal info.
Click to expand...

And my laziness at having to put in my details every time will probably mean I buy less on impulse.
 
Visualante2 said:
So because Facebook is free we shouldn't care when they leak your private communications? Ridiculous argument.
Click to expand...

Absolutely this is not what I wrote. I only stated in Neogaf the only thing you put in is your email. Of course you should be worried if something leaks on Facebook, since on FB are stored a lot of infos (and yes, even if it's free, when you do microtransactions or similar in games, you put your CC card info here, I believe, so in a way is similar to Steam: is totally free except when you want to buy something, and that's the time where you store sensible infos in FB, because only your name and mail are mandatory in FB, but your name is visibile to all people no matter what if you decide yo use is).

I wrote shit can happen, but don't need to put your CC, phone number, real name, addrees etc. In a plan is "free" like Steam, with the excepton when you buy something on Steam you are forced to put your real info in it for tax purpose.

So the moment I pay, I demand maximum security. I repeat myself, shit can happen, hackers can do nasty thing. In that case, I wouldn't argue.
But this was Valve's fault, and they acted badly and their statement is insulting.
 
PetriP-TNT said:
If I make a statement about an issue I caused I usually start the timing when the information reaches me to the point where the fix has been shipped. When splitting hairs like 30 minutes it's always important to remember that rolling out things takes time and thats how it is. Would the statement made you feel safer if it laid out things by the minute? Maybe.
Click to expand...

It's not that it's 30 minutes or splitting hairs. It's that for a good 2 and something hours, there's radio silence on Steam's end for such a widespread problem.

If you leave people in the dark for a long time, without a word or otherwise, during a time period where you can randomly view other people's information, it behooves you to drum up something.

There's nothing wrong with the time period it takes to fix the problem. You gotta do what you gotta do, regardless of how bad the fuck up is. But there's no way Valve didn't see that shit for hours.
 
TheSpoiler said:
It's not that it's 30 minutes or splitting hairs. It's that for a good 2 and something hours, there's radio silence on Steam's end for such a widespread problem.

If you leave people in the dark for a long time, without a word or otherwise, during a time period where you can randomly view other people's information, it behooves you to drum up something.

There's nothing wrong with the time period it takes to fix the problem. You gotta do what you gotta do, regardless of how bad the fuck up is. But there's no way Valve didn't see that shit for hours.
Click to expand...

They should turn off the servers immediately. They didn't.
It would ake hours/days to fully restart servers? No matter, security is more important then everything in my opinion.
They could extend the sale or whatever, that is inexcusable.

xPaw said:
I'll speak on behalf of SteamDB here. We always tweet and post information on our blog only after verifying it ourselves, we didn't base our information on speculation, but rather on our own research. It was pretty clear that it was a caching issue, as it was caching the first non-cache hit on any url (you could easily test this by adding ?something=random to the url, and then loading it from another browser/machine while not being logged in).

I believe this was caused by a misconfiguration on Akamai (Valve's CDN), which caused this to happen. You can use Akamai's debug headers to get some interesting information, and during the issue it was showing X-Check-Cacheable: YES at all times, which is not good. There are no indications that anything could be done on your behalf (caching is read-only).

Yes, we know we know we shouldn't be speaking for Valve, but we find it more important to keep users aware of the issue. Our track record is pretty damn good in that sense, and if the response Gamespot/Kotaku received from Valve is indeed correct, we pretty much nailed our caching theory.

Before publishing the blog post, we had multiple people that work closely on Steam (besides us) proof read it too.
Click to expand...

Thanks for the clarification and the time you spent on doing the post (plus the job you guys do on steam DB). :)
 
saunderez said:
It's got nothing to do with stress testing, if the caching issue occurred at all it would've presented with 5 users let alone 5 million. They didn't do their due diligence and in the process user data was leaked.
Click to expand...

Not if there's a bit width increase that causes the issue. For example: a system could work well for 16 millions concurrent but fail for 17 million concurrent users, since the 24 bit limit would be approached (under such circumstances, an ID for the 17,000,000 would be shared/identical with the ID for 222,784).

At any rate, making any kind of configuration change during Christmas Day is a bad idea. And now, failing to inform the customers of what caused the issue (when the issue is as severe as this) is an even worse idea.
 
Ludens there are people on this website who leak insider information in an anonymous capacity. Some of them are vetted by moderators. If that information were to get out into the public domain then people could potentially lose their jobs. I don't know perhaps people are vetted with off site e.mails but to me that seems reason enough to have responsible security measures.

Your whole concept of paid vs. free services remains flawed in my opinion. Security should be based on what there is to be lost. That can be information, reputational damage, all sorts of things that goes beyond what fields or financial relationships the organisation and users have.
 
Can someone help me clear some confusion: did or did not Valve shut down Steam at any point? If yes, for how much time? I'm trying to pinpoint when exactly the problem began and when it ended. I'm reading on some info sites that Valve did shut down Steam, and I'm pretty sure people here said that Valve never shut down Steam. I'm wary of misinformation.
 
Nohar said:
Can someone help me clear some confusion: did or did not Valve shut down Steam at any point? If yes, for how much time? I'm trying to pinpoint when exactly the problem began and when it ended.
Click to expand...
Steam was down for "routine maintenance" for me for a while around that time. Or so my friends list on the client told me. After I read the thread I assumed they had turned some things off until it was fixed. I'd say probably an hour if not more.
 
Visualante2 said:
Ludens there are people on this website who leak insider information in an anonymous capacity. Some of them are vetted by moderators. If that information were to get out into the public domain then people could potentially lose their jobs. I don't know perhaps people are vetted with off site e.mails but to me that seems reason enough to have responsible security measures.

Your whole concept of paid vs. free services remains flawed in my opinion. Security should be based on what there is to be lost. That can be information, reputational damage, all sorts of things that goes beyond what fields or financial relationships the organisation and users have.
Click to expand...

You misurunderstand me. Security is ALWAYS the FIRST thing. What I want to say is this: Origin, Valve, GOG, Uplay are COMMERCIAL site, you share MORE info compared to Facebook or Neogaf. On FB you can share where you live (why? It would be stupid in my opinion, but the point is this info is up to you to be shared. It's not forced.) or other infos, but it's up to you. If something leaks, worst case scenario is you get a ton of spam (if you only put your mail in and you never bought anything, of course).
Same on Neogaf.
With this I don't mean Neogaf, Facebook or any other free service should be an open dock. I'm only saying if something leaks on Steam/PSN, it's a huge damage involving both money and infos related to your persona (where you live, phone number etc), so those service should be armored. Hackers do nasty things? IT CAN HAPPEN. What I hate is the fact Steam itself fucked up, not hackers.

Said this, ALL services should be protected as much as possible, no matter if they are free or they aren't. But in case of commercial services (which involve your personal data in order to work properly, a thing you can't avoid), I don't expect you fuck up so badly, even because you are in the market from ten years. I don't mind if it's christmas or whatever, if you have problems like only a bunch of people to control Steam on x-mas, DON'T RUN THE SALE during that perdiod. Do that when enough people can control every worse case scenarios.
 
Nohar said:
Can someone help me clear some confusion: did or did not Valve shut down Steam at any point? If yes, for how much time? I'm trying to pinpoint when exactly the problem began and when it ended. I'm reading on some info sites that Valve did shut down Steam, and I'm pretty sure people here said that Valve never shut down Steam. I'm wary of misinformation.
Click to expand...

I think community side was up the whole time. They shutdown the store though. Based on a post I made the store came back up around 1 am (gmt+2). Don't know what functionality was available at that time but I was able to access the store.
 
Kathian said:
This was not an attack. And its more the response. They caused it and reacted both slowly and flippantly.
Click to expand...

Ludens said:
What is really shitty is their statement: they didn't apologize, they took this like nothing. This is infuriating and disrispectful toward their customers, plain and simple.
Click to expand...


I'm okay with them not apologising. I care more about it not happening again. I wonder if they have good reasons for not doing it.
Also, I would never take a company apology at face value. That's just PR speak. I find most apologetic releases to be insincere and really have a brush off rhetoric. I'd rather people hold them up on it instead of being outraged over a lack of a PR release (which doesn't mean much).
 
Visualante2 said:
Steam was down for "routine maintenance" for me for a while around that time. Or so my friends list on the client told me. After I read the thread I assumed they had turned some things off until it was fixed. I'd say probably an hour if not more.
Click to expand...

kAmui- said:
I think community side was up the whole time. They shutdown the store though. Based on a post I made the store came back up around 1 am (gmt+2). Don't know what functionality was available at that time but I was able to access the store.
Click to expand...

Thank you very much for your answers!
 
PetriP-TNT said:
If I make a statement about an issue I caused I usually start the timing when the information reaches me to the point where the fix has been shipped. When splitting hairs like 30 minutes it's always important to remember that rolling out things takes time and thats how it is. Would the statement made you feel safer if it laid out things by the minute? Maybe.
Click to expand...
No, it's called honesty. If you claim an issue was live for "less than an hour" when there's public evidence from others affected by the issue well before that time, I'm going to call bullshit on your claim.

What Valve should have said: "An issue occurred for X hours which resulted in users being able to access others' personal information. Full credit card details were not leaked, nor were any payment details. Unauthorised purchases could not be made. Full names could be accessed. Address data, telephone numbers and the last four digits of stored card numbers may have been viewable. We sincerely apologise for the inconvenience this has caused our users, and we want to reassure them that the problem is now fully resolved, and no action is required by users. We will be taking immediate steps to ensure an issue such as this does not happen in the future. Full details of the security measures we will take to ensure this will be posted in due course once they are finalised. Should you have any concerns regarding your data, please do not hesitate to contact us and we will do all we can to assist."

What Valve actually said: "Steam is back up and running without any known issues. As a result of a configuration change earlier today, a caching issue allowed some users to randomly see pages generated for other users for a period of less than an hour. This issue has since been resolved. We believe no unauthorized actions were allowed on accounts beyond the viewing of cached page information and no additional action is required by users."
 
Vigilant Walrus said:
I'm okay with them not apologising. I care more about it not happening again. I wonder if they have good reasons for not doing it.
Also, I would never take a company apology at face value. That's just PR speak. I find most apologetic releases to be insincere and really have a brush off rhetoric. I'd rather people hold them up on it instead of being outraged over a lack of a PR release (which doesn't mean much).
Click to expand...

Yeah, but they didn't mention your personal infos can be seen by any random dude around the Internet.
 
Good for Valve that they were the ones spearheading the return of PC games. No way they'd own the market with that shitty ass service without the headstart.
 
I have got into my account, everything seems ok, I had no payment info on my account anyway, I don't have my phone number in there either, would it be advisable for me to put my phone number in for verification reasons? I don't use my PC/Steam for games very much so I don't know if it's worth it, but then again if something like this does happen again I don't want to lose access to my Steam account.

Advice/thoughts/views please.
 
B1G_A1AN said:
I have got into my account, everything seems ok, I had no payment info on my account anyway, I don't have my phone number in there either, would it be advisable for me to put my phone number in for verification reasons? I don't use my PC/Steam for games very much so I don't know if it's worth it, but then again if something like this does happen again I don't want to lose access to my Steam account.

Advice/thoughts/views please.
Click to expand...
I think putting your phone number for 2 way verification and/or for emergency recovery is worth it.. worst scenario you deal with some prank. But you have extra method to recover your account.
 
Yeah, one of the bigger things that bothers me is the lack of care that went into the explanations. You owe it to your consumer to be honest and upfront about the problems and issues to better arm themselves.

If I read that statement, I'd have never known my personal information and last digits of my card were open to anyone in the damn world. That's completely inexcusable.
 
xPaw said:
I'll speak on behalf of SteamDB here. We always tweet and post information on our blog only after verifying it ourselves, we didn't base our information on speculation, but rather on our own research. It was pretty clear that it was a caching issue, as it was caching the first non-cache hit on any url (you could easily test this by adding ?something=random to the url, and then loading it from another browser/machine while not being logged in).

I believe this was caused by a misconfiguration on Akamai (Valve's CDN), which caused this to happen. You can use Akamai's debug headers to get some interesting information, and during the issue it was showing X-Check-Cacheable: YES at all times, which is not good. There are no indications that anything could be done on your behalf (caching is read-only).

Yes, we know we know we shouldn't be speaking for Valve, but we find it more important to keep users aware of the issue. Our track record is pretty damn good in that sense, and if the response Gamespot/Kotaku received from Valve is indeed correct, we pretty much nailed our caching theory.

Before publishing the blog post, we had multiple people that work closely on Steam (besides us) proof read it too.
Click to expand...

Thank you for sharing this, and I want to tell you that you did a great job at communicating to the users where Valve did not during this.

Is there any way of telling who was affected based on your knowledge about the caching system? I have my doubts that Valve will come forward at this point with any further information so customers can safely rest knowing they were not affected or take necessary security measures if they were.
 
I don't think that this will have any lasting effect on Valve's reputation. It is massively shielded and inflated by fanboys on the internet.

I understand that some problems and accidents happen, no company is immune to human error or a coordinated attack. However their attitude at exposing billing information is staggering. They need to figure out which accounts had their billing information leaked and inform those people or state that EVERYONE's data might have been exposed.
 
Dunkley said:
Thank you for sharing this, and I want to tell you that you did a great job at communicating to the users where Valve did not during this.

Is there any way of telling who was affected based on your knowledge about the caching system? I have my doubts that Valve will come forward at this point with any further information so customers can safely rest knowing they were not affected or take necessary security measures if they were.
Click to expand...

The only thing I can think of is, people that actually visited account/checkout pages were potentially affected by private information leakage.
 
Had this been Sony or Microsoft, people would have gone batshit insane, right now.

I guess I will have to buy prepaid cards for Steam just like I do for other services from now on.
 
Labadal said:
Had this been Sony or Microsoft, people would have gone batshit insane, right now.

I guess I will have to buy prepaid cards for Steam just like I do for other services from now on.
Click to expand...

Having a Steam Wallet balance would have been about as open to abuse in this instance as a saved payment method. Just don't save payment methods to your account. Yes, it's a little annoying having to type in your billing info each and every time you buy something, but better that than being out of pocket.
 
TheSpoiler said:
Yeah, one of the bigger things that bothers me is the lack of care that went into the explanations. You owe it to your consumer to be honest and upfront about the problems and issues to better arm themselves.

If I read that statement, I'd have never known my personal information and last digits of my card were open to anyone in the damn world. That's completely inexcusable.
Click to expand...

Yeah, there's nothing on the main page and nothing on steam itself. You have to dig to find any reason for what just happened.

I at least woke up to a needing to re-verify my email address, so there's that. Not sure if anyone else had to or if that means someone tried to login?
 
If you are really lazy, like me, you can use LastPass form fill feature to store credit card info. One click fill in via the browser store.
 
You must log in or register to reply here.

Similar threads

PSYCHO DEAD | Gameplay Reveal Trailer (Third-person survival horror) [PS5, Steam]
23
147
Macattk15 replied
  • Poll
Steam users threaten to boycott Metal Eden for using AI voices
News Drama  2 3
117
1K
rkofan87 replied
Digimon Story Time Stranger debuts with over 70k concurrent users on Steam (~11x the launch CCU of Digimon Survive)
Sales-Age 
5
87
Liopleurodon replied
Steam hits 40m concurrent users.
3 4 5 6 7
300
16K
rodrigolfp replied
[The Verge] Xbox Ally pricing and preorders info coming today. [Update] Prices revealed: $999 Z2 Extreme; $599 Z2
7 8 9 10 11
544
1K
Duchess replied
Top Bottom