Support NeoGAF

[RadioactiveLobster]

So the official response is basically "Oops, our bad."

No apology for the thousands of email addresses that were stumbled upon.

No remorse for revealing entire purchase histories.

No breakdown of why the hell they would plan to do this on Christmas day.

No promise to not pull this stupid shit again.

It's almost like people have forgotten that time exists and progresses. There is this thing called "tomorrow" and chances are, we'll hear something then. Just like the last time a breach occurred and they released information the following day after they had time to investigate the issue.

 

[Herpes Reasons]

It's almost like people have forgotten that time exists and progresses. There is this thing called "tomorrow" and chances are, we'll hear something then. Just like the last time a breach occurred and they released information the following day after they had time to investigate the issue.

People want heads on a platter today and won't rest until they get it.

 

[Palculator]

It's almost like people have forgotten that time exists and progresses. There is this thing called "tomorrow" and chances are, we'll hear something then. Just like the last time a breach occurred and they released information the following day after they had time to investigate the issue.

But we could be being unreasonable right now.

 

[ArkhamFantasy]

Out of curiosity, what is common procedure for a software company that detects a security threat to its customers?

Will they allow a programmer who detects a serious threat to shut down a website/service just to be on the safe side or is there alot of debate between programmers/higher ups before they make that kind of decision?

 

[KINGofCRA5H]

Quick summary:

• Viewing the Account Details page on Steam showed you as logged in as another user and allowed you to see their account details, including amongst other things the last 4 digits of their card number, their Steam account name and the E-Mail the account is attached to.
• People start testing it and realize it does work; people become nervous about being affected and their account information being shown.
• Couple of people report there have been purchases made causing people who have attached payment methods to panic
• Huge demand that Valve takes down the servers and criticism for taking so long to do so.
• SteamDB offers their theory on what happened, says it's not safe to log in or even view Steam pages making those who have been checking if the issue was still ongoing even more nervous
• "Yeah yeah we're working on it" statement made by a Community Manager on Steam, declining a hacking attack
• 1 hour later Steam servers go down (finally)
• It is revealed that you could also see people's addresses and their full phone number(s) due to this issue. (if saved to the account due to the payment option)
• Servers come back up without Valve saying anything
• Valve releases short non-apology confirming SteamDB's theory but doing jack to inform people about their personal information being exposed.
• Discussion about Valve's handling of the situation mixed with a bunch of people coming in thinking it's still an ongoing issue

I think that's the gist of the thread, sorry if I missed something.

Probably should go in the OP

 

[direct_quote]

So is this over now? Can I log onto steam?

 

[GrandiaX]

It's almost like people have forgotten that time exists and progresses. There is this thing called "tomorrow" and chances are, we'll hear something then. Just like the last time a breach occurred and they released information the following day after they had time to investigate the issue.

Derail. Who is that hot girl in your user avatar?

 

[pahamrick]

If anyone doesn't have a strong unique password for their email, they might want to get one ASAP.

Went to check my email a bit ago, got wrong password error. My email is through my cable company, so I called them up to see what the issue was because I tried three times and it failed with the wrong password error each time.

According to the rep, my email was accessed enough times with the wrong password that it locked the email account. She had to reset the password and unlock it so I could change the password and regain access.

 

[Disaster Nebraska]

Isn't this splitting hairs? If Valve screwed up by misconfiguring Varnish so that it cached account pages when it shouldn't have been, that's still a "caching issue." No one's saying that "caching issue" means Valve is somehow off the hook or that Akamai/Varnish/etc. are somehow at fault.

Then let me explain again:

I don't believe it was a caching issue at all. This isn't how caches work, and you don't just "misconfigure" one to act this way. There's still the very basic matter of how servers work. For instance, despite hundreds if not thousands of IP addresses connecting to a server, you notice how you don't, oh I dunno, get randomly redirected to a page that someone else was viewing or to their account page? I mean, I understand today's events contradict that, but stay with me.

The reason why is because each server since the 80s knows how to keep a session and not randomly pass the results of requests from IP A to the user at IP B. This is incredibly basic stuff and has almost nothing to do with how a CDN or caching service works on a web server. The cache doesn't say "oh, the user at IP A loaded up their account, but I guess I'll just randomly give that same data to the user at IP B because [handwave] a "caching misconfiguration" was to blame."

Now, if someone with knowledge of web servers would like to correct me, I could certainly be wrong. I only know so much, and maybe it was a caching issue?

Well, then we're still in a pickle even if it's actually a caching issue. See, I've worked with Akamai (a CDN) and Varnish (a software cache) and ive never seen either of them haphazardly misconfigured to the point of a customer data breach. Ever. Usually they will simply fail to serve content and then the web page will show as "down". Or, if a portion of the content is on a CDN (which is how CDNs are usually used) then that content will fail o show up. This would result in -- for example -- your website loading but certain images not loading on the page.

So, if this was due to "caching" it can mean two things, both of them bad and both of them are Steam's fault:

One possibility is that Steam was storing unencrypted session data or account data on their CDN and/or through Varnish. This is pretty much unnecessary. Customer info is stored in a database, not as a large flat file like an image. There's no performance-based reason to store 20kb (yes, kilobyte) snippets of data in a cache, and especially not in an external CDN.

You'd actually want to use caching with the database engine. I don't know if they're using MariaDB or MySQL (Linux side) or MSSQL (Windows) but all of these offer caching options specifically aimed at speeding up database queries. No amount of misconfiguration would cause a query result to
get passed to another IP.

If it really is caching, it means that Valve is running one of the least-secure storefronts on the Internet. They'd have to fuck their server configurations so bad and have such terrible
website code to allow for something like this to happen.

Or it's a hack.

 

[Aselith]

People want heads on a platter today and won't rest until they get it.

And rightly so. This is exactly what you establish emergency contacts for and it seems like Valve sent an intern down to work on the servers and zip off a terse reply to Kotaku. People just want some assurance that the issue is being taken seriously and all indications point to the opposite.

 

[RadioactiveLobster]

Derail. Who is that hot girl in your user avatar?

Mia Sollis, Bing image search it. Thank me later. There is also a subreddit.

 

[Gentleman Jack]

It's almost like people have forgotten that time exists and progresses. There is this thing called "tomorrow" and chances are, we'll hear something then. Just like the last time a breach occurred and they released information the following day after they had time to investigate the issue.

Of course, people still want to safely buy and play video games today and are understandably not sufficiently mollified by "We believe no unauthorized actions were allowed on accounts beyond the viewing of cached page information and no additional action is required by users." Yes, it's not the end of the world to not feel safe using Steam until tomorrow. But it's still an unnerving amount of nothing coming during a time when people just want to relax and enjoy a holiday in what manner they so choose.

 

[gogosox82]

Quick summary:

• Viewing the Account Details page on Steam showed you as logged in as another user and allowed you to see their account details, including amongst other things the last 4 digits of their card number, their Steam account name and the E-Mail the account is attached to.
• People start testing it and realize it does work; people become nervous about being affected and their account information being shown.
• Couple of people report there have been purchases made causing people who have attached payment methods to panic
• Huge demand that Valve takes down the servers and criticism for taking so long to do so.
• SteamDB offers their theory on what happened, says it's not safe to log in or even view Steam pages making those who have been checking if the issue was still ongoing even more nervous
• "Yeah yeah we're working on it" statement made by a Community Manager on Steam, declining a hacking attack
• 1 hour later Steam servers go down (finally)
• It is revealed that you could also see people's addresses and their full phone number(s) due to this issue. (if saved to the account due to the payment option)
• Servers come back up without Valve saying anything
• Valve releases short non-apology confirming SteamDB's theory but doing jack to inform people about their personal information being exposed.
• Discussion about Valve's handling of the situation mixed with a bunch of people coming in thinking it's still an ongoing issue

I think that's the gist of the thread, sorry if I missed something.

Thanks for the info. Been away all day and just now reading about all this. I didn't have any money in my account but I did have my debit card and email on file. I went ahead and changed my password to my email and deleted my debit card from the account. Will call my bank about it in the morning just in case. This is a pretty big screw up by valve. Why do these changes on Christmas of all days? Literally makes no sense.

 

[faceless007]

It's almost like people have forgotten that time exists and progresses. There is this thing called "tomorrow" and chances are, we'll hear something then. Just like the last time a breach occurred and they released information the following day after they had time to investigate the issue.

I don't think people are necessarily expecting a full minute-by-minute breakdown of exactly how the breach happened to be posted today. However, it took at least 3-4 hours for anyone at Valve to even publicly acknowledge the issue, and they only did so as comments to the handful of journalism outlets that bothered to keep pestering them for comments, instead of prominently and immediately on their own damn web site, and the comment itself was worryingly nonspecific and almost brusque in acknowledging seriousness of what happened. (But they were very quick to get the actual storefront up and running again as soon as they could with no acknowledgement of the issue, wasn't that considerate of them?) To just quote charlequin:

Yep, this is beyond bush league. The most basic responsibility of a service-based company is to announce that something has happened, share any confirmed info, and shut down service until it can be verified to be safe. Valve has done none of that.

I hope people who praise valve's management structure take a good look at this situation. Every part of this fiasco -- the half-assed ddos mitigation, the apparent untested launch of code with a massive security hole, and the complete silence to their customer base -- is a direct result of an organizational culture with no leadership, no responsibility, and no employees who are expected to do difficult or unpleasant work.

 

[draetenth]

Periodic reminder that GOG did a stunt that involved a fake shutdown of their service with all content reported to be lost as part of an idiotic promotion of their DRM-free games.

But, that's just it: it was just an incredibly idiotic pr move by GOG. It wasn't a colossal security blunder like this is with Steam.

The bigger issues should be that afaik GOG still doesn't have 2-step authorization and GOG is apparently extremely picky about the games that they allow (they apparently turn down games because... reasons).

Really, GOG should fix those two issues and take advantage of security flaws like this. This is the time when we need things like GOG/Origin/Uplay and anyone else to put heat on Valve.

 

[Palculator]

Mia Sollis, Bing image search it. Thank me later. There is also a subreddit.

Oh my...

 

[anaron]

People want heads on a platter today and won't rest until they get it.

Rightfully so. For fucks sake, this is private personal information.

 

[Hidden One]

Has everything been worked out yet?

 

[Disaster Nebraska]

Has everything been worked out yet?

Lots of misinformation getting passed around and all Valve has said that "it happened, but nothing got compromised", which we already know is false because the evidence is right in front of us.

 

[StereoVsn]

But, that's just it: it was just an incredibly idiotic pr move by GOG. It wasn't a colossal security blunder like this is with Steam.

The bigger issues should be that afaik GOG still doesn't have 2-step authorization and GOG is apparently extremely picky about the games that they allow (they apparently turn down games because... reasons).

Really, GOG should fix those two issues and take advantage of security flaws like this. This is the time when we need things like GOG/Origin/Uplay and anyone else to put heat on Valve.

Being picky isn't really an issue, if they want to curate their storefront (for whatever reason), it doesn't matter. Absense of two-factor auth though is very puzzling and unprofessional in this day and age. However as Valve amply just demonstrated, two factor auth can't stop server side fuckups.

I am really disappointed in Valve. Between their atrocious support structure, and apparently terrible release code QA as well as security issues it's frankly a bit scary to think that PC gaming is basically dependent on a company who couldn't give two flying fucks about its customers.

 

[GuardianE]

Valve's response is completely unacceptable at this point.

 

[SillyJoe]

Give me my free game.

 

[Palculator]

Lots of misinformation getting passed around and all Valve has said that "it happened, but nothing got compromised", which we already know is false because the evidence is right in front of us.

They didn't say it wasn't compromised, they said "no unauthorized actions were allowed on accounts."

 

[RootCause]

At least one good thing happened because of this, it got me to check on my super old email I use for Steam

Lol

You got this notification through email? I'd like to know if anyone attempted to get in my account.

 

[Astarte]

Give me my free game.

Free copies of retail TF2 for everyone

 

[Asgaro]

Quick summary:

• Viewing the Account Details page on Steam showed you as logged in as another user and allowed you to see their account details, including amongst other things the last 4 digits of their card number, their Steam account name and the E-Mail the account is attached to.
• People start testing it and realize it does work; people become nervous about being affected and their account information being shown.
• Couple of people report there have been purchases made causing people who have attached payment methods to panic
• Huge demand that Valve takes down the servers and criticism for taking so long to do so.
• SteamDB offers their theory on what happened, says it's not safe to log in or even view Steam pages making those who have been checking if the issue was still ongoing even more nervous
• "Yeah yeah we're working on it" statement made by a Community Manager on Steam, declining a hacking attack
• 1 hour later Steam servers go down (finally)
• It is revealed that you could also see people's addresses and their full phone number(s) due to this issue. (if saved to the account due to the payment option)
• Servers come back up without Valve saying anything
• Valve releases short non-apology confirming SteamDB's theory but doing jack to inform people about their personal information being exposed.
• Discussion about Valve's handling of the situation mixed with a bunch of people coming in thinking it's still an ongoing issue

I think that's the gist of the thread, sorry if I missed something.

 

[charlequin]

If it is supposedly a "caching issue" (I doubt it is) there really isn't some configuration that you just make an oopsie and it randomly exposes people's information. I have worked both with Akamai and Varnish (the two services that at this point were identified as the source of the supposed "caching issue") and this isn't how either of them work.

It's definitely a cache-related issue, either internal to their own software or in a reverse-proxy layer somewhere. This type of behavior is what you see if either you break the cache keying somehow so that it's not based on the unique identifier of the record (or you leave it out altogether) and every time the TTL runs out the next poor sap to load a cached page gets their info stored and sent out to everyone.

I would agree with you that it's almost certainly not a CDN-level caching issue, but that's not the only layer of caching involved in an app like theirs.

this is absolute spot on. this is the result of a culture of not putting customers first. there's this expectation, especially by people just now entering the workforce, of never having an unpleasant day at work and never having to put in effort. We need month long vacations at a moments notice because work should never come before your personal life.

These don't necessarily have anything to do with each other. I've worked at a tech company that had an unlimited vacation policy, but still had quick response to breaches and strong customer service, because people had clear responsibilities, were trained to do them correctly, and were motivated by the way management operated to actually do their jobs well.

Honestly, companies that have good work/life balance and usually don't force people to work extra hours tend to be much better about responding to this type of issue, since employees will recognize the severity of a scenario that actually requires work outside the norm and will generally be going into an emergency situation with much higher morale.

 

[Fury451]

So the official response is basically "Oops, our bad."

No apology for the thousands of email addresses that were stumbled upon.

No remorse for revealing entire purchase histories.

No breakdown of why the hell they would plan to do this on Christmas day.

No promise to not pull this stupid shit again.

So a Valve response then. Good grief, what a mess.

I imagine they will release a more detailed breakdown soon, but there's little comfort in having to wait to find out exactly what it is that happened.

 

[kevin1025]

I would like Ronin and 24 cards as compensation.

 

[ShockingAlberto]

Valve handled this about as poorly as anyone could have.

I don't think they can continue having a company structure where people only do customer service and community management when they want to.

 

[arts&crafts]

How the hell do you sort steam by "items on sale" and "lowest price" I am sick of 20 pages of f2p games showing up first. Just realized this was the wrong, wrong steam thread.

 

[uncredited male]

Give me my free game.

You will get L4D2 and like it.

 

[Enter the Dragon Punch]

i still don't understand why valve continues to purposefully skimp out on support services. they're some of the worst i have ever, ever, had to use regardless of what kind of service a company provides

 

[Aselith]

I would like Ronin and 24 cards as compensation.

If you get anything, it will be like Left 4 Dead 2 and Portal 2.

 

[hardcastlemccormick]

So the official response is basically "Oops, our bad."

No apology for the thousands of email addresses that were stumbled upon.

No remorse for revealing entire purchase histories.

No breakdown of why the hell they would plan to do this on Christmas day.

No promise to not pull this stupid shit again.

This is all bare minimum stuff for a security breach. Baffling.

It's almost like people have forgotten that time exists and progresses. There is this thing called "tomorrow" and chances are, we'll hear something then. Just like the last time a breach occurred and they released information the following day after they had time to investigate the issue.

Time indeed progresses - a lot can happen to sensitive information between now and tomorrow. If there's information out in the wild, people need to know now. Tomorrow is too late.

 

[kevin1025]

If you get anything, it will be like Left 4 Dead 2 and Portal 2.

Half-Life: Source or bust!

 

[StereoVsn]

Valve handled this about as poorly as anyone could have.

I don't think they can continue having a company structure where people only do customer service and community management when they want to.

Sure they can, unless they run afoul of a federal regulation of some sort (either US, CA or EU). There is no good alternative and everybody is back to buying games (perhaps removing stored payment info).

 

[Mivey]

i still don't understand why valve continues to purposefully skimp out on support services. they're some of the worst i have ever, ever, had to use regardless of what kind of service a company provides

They have hired professors of robotics, machine learning and psychology to build an army of robots to staff their support division. They should be done in the next few decades. Alternatively, if the project doesn't pan out, Valve could use its remainders to simply take over the world and force a few countries to do it for them . I would suggest the UK, no one would understand them, but at least they're polite.

 

[Aselith]

Half-Life: Source or bust!

You also get a special Dota 2 immortal commemorating the breach for Dark Seer (it has no special effect, just gives him a villain mustache) and a Team Fortress 2 key.

 

[Inspector Jones]

Not as outraged as others; but I can't deny that Valve is slow as shit. Also, their customer support is still the worst.

I expect a full statement tomorrow on Steam.

 

[Fireblend]

That "statement"... smh

To be fair though, it's probably in Valve's best interest to downplay it as much as possible. I really hope they get called out for their shit though, what we saw today is inexcusable.

 

[charlequin]

but from where i am standing , it seems more logical to , when faced with a situation such as this to first fixing wathever caused it Asap to prevent any further users private info being visible to everyone.

I agree, but if there isn't a big red button you can hit to fix the problem then the way to do that is to bring the service down completely and investigate.

So the question is why in the world is Valve seeding non encrypted data into CDNs?

CDNs have access to an unencrypted version of any data they serve out, they don't just forward pre-encrypted data along.

If this were an issue on the CDN level (and again, I don't think it was) it would've been as a result of fucked-up Cache-Control headers, which led to the CDN caching pages that should never be cached.

 

[StereoVsn]

That "statement"... smh

To be fair though, it's probably in Valve's best interest to downplay it as much as possible. I really hope they get called out for their shit though, what we saw today is inexcusable.

Just imagine if this happened to Microsoft, Sony, Nintendo, EA, or Ubisoft? Can you just see this exploding everywhere? Valve does this, meh, it will blow over.

 

[JaseC]

How the hell do you sort steam by "items on sale" and "lowest price" I am sick of 20 pages of f2p games showing up first. Just realized this was the wrong, wrong steam thread.

http://store.steampowered.com/searc...ce_ASC&category1=998&os=win&specials=1&page=1

 

[StereoVsn]

I agree, but if there isn't a big red button you can hit to fix the problem then the way to do that is to bring the service down completely and investigate.


CDNs have access to an unencrypted version of any data they serve out, they don't just forward pre-encrypted data along.

If this were an issue on the CDN level (and again, I don't think it was) it would've been as a result of fucked-up Cache-Control headers, which led to the CDN caching pages that should never be cached.

They shouldnt have caching account info though since that's barely any traffic one would think. Like game pictures/screens/descriptions even maybe, but account info should have only been pulled on demand. I mean, I have no clue how Valve does it but I've worked with some online systems running off AWS and utilizing various CDNs and I've never seen account info being cached. That's what's puzzling to me.

 

[Disaster Nebraska]

It's definitely a cache-related issue, either internal to their own software or in a reverse-proxy layer somewhere. This type of behavior is what you see if either you break the cache keying somehow so that it's not based on the unique identifier of the record (or you leave it out altogether) and every time the TTL runs out the next poor sap to load a cached page gets their info stored and sent out to everyone.

I would agree with you that it's almost certainly not a CDN-level caching issue, but that's not the only layer of caching involved in an app like theirs.

You say "if you break the cache keying somehow" like it's a normal thing. For that to happen, it would be a failure on multiple points in the server, not just caching. There's still the basic security and redundancies that prevent session data or request results being passed from one requesting IP to a completely different one.

No, this is definitely NOT normal behavior that just happens -- oopsie -- due to a misconfiguration or a simple code push.

 

[arts&crafts]

http://store.steampowered.com/searc...ce_ASC&category1=998&os=win&specials=1&page=1

Thanks, it wasnt working for me..

 

[BlackJace]

Methinks it's probably time for Valve to rethink their whole "free structure, collective agenda" angle in regards to how the company is run.

Shit, even outsourcing a customer service base would be better than their current attitude towards CS, no?

 

[StereoVsn]

You say "if you break the cache keying somehow" like it's a normal thing. For that to happen, it would be a failure on multiple points in the server, not just caching. There's still the basic security and redundancies that prevent session data or request results being passed from one requesting IP to a completely different one.

No, this is definitely NOT normal behavior that just happens -- oopsie -- due to a misconfiguration or a simple code push.

Could have been a large Auth package update. Of course why would you push an update like that during very busy shopping time, I don't know.

 

[Usobuko]

Valve handled this about as poorly as anyone could have.

I don't think they can continue having a company structure where people only do customer service and community management when they want to.

Valve is the kind of company I will buy if it's publicly traded, at a premium over the average financial ratios of the index.

Their approach is less of being stubborn and more of being greedy. And they get away all the time. Any misstep will be forgotten and relegated to Meme-esque jokes.

A hallmark of a great brand.