poppabk
Cheeks Spread for Digital Only Future
But somewhere that guy exists. The guy who pushed the final button that sent this out. He is probably curled up in a ball somewhere though, doubt he is having a laugh about it.
-
Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.
Worldwide Microsoft Outage: Flights Grounded, Sky News Off Air, Workplace Systems Down
- Thread starter AV
- Start date
j0hnnix
Member
If he was real my old company should hire him. They love pushing patches with little to no testing to meet perceived client deadlines.
Last edited:
Bitmap Frogs
Mr. Community
Testing on production is a long standing tradition
j0hnnix
Member
Its the only true real world test. Lol
Bitmap Frogs
Mr. Community
A tester tests a bar:
Asks for a parakeet
Asks for nothing
Asks for a beer
Declares the bar tested
The next day the first customer asks where the bathroom is and the bar explodes.
*im qa
ForAcademicPurposes
Member
When our virtual money gets wiped out that is when shit will truly hit the fan. Blade Runner 2049 is a documentary.
IntentionalPun
Ask me about my wife's perfect butthole
Some of the biggest systems in the world don’t have much of a choice due to the sheer size and complexity.
But that’s why we have extensive feature flag systems, a/b testing, regional rollouts, etc.
Shadowstar39
Member
you can blame microsoft for that and companies wanting someone else to do all their server and email stuff. Oh you will take care of it if we pay you a monthly sub, sure , no multiply that by millions across the globe. There is a reason ms was able to buy activsion and others without breaking a sweat in finances. Companies gave over their it to them
Last edited:
Zathalus
Member
Microsoft is not even the market leader in cloud services. Nor does this outage have anything to do with infrastructure in the cloud. It’s an update pushed to end users by a antivirus provider, you can have all the affected servers in your local datacenter and you would still be hit.
Unknown Soldier
Member
LOL CrowdStrike really working hard to earn the trust of their 24,000 customers today, trying to be the Boeing of cyber security I guess
poppabk
Cheeks Spread for Digital Only Future
It's kinda scary that their stock is still up 25% from the start of the year even after this colossal fuck-up.
ReBurn
Gold Member
Yep. My experience today has been that VM's running on VMWare in my datacenter, VM's running in the cloud, and on-prem servers running on dedicated hardware were taken down by the CrowdStrike issue. When it came to fixing it the cloud backups were easier to restore than the ones I had to do on-prem.
I can imagine how much it would have sucked if I also still had on-prem exchange and sharepoint servers in the mix. I don't see how anyone actually thinks that's better than cloud-based options right now.
ReBurn
Gold Member
Buy it while it's low because it will go back up.
BadBurger
Many “Whelps”! Handle It!
Switching to Exchange Online alone saves us dozens of hours of labor every few months compared to on-prem - which was expensive and sucked to maintain and work on. Microsoft 365, especially AVD/WVD, was a god send as well.
ReBurn
Gold Member
A lot of the time saved is late nights and weekends because you can really only maintain some of those systems when everyone else is sleeping. Cloud-based services improve overall quality of life for my network and security admins in so many ways.
BadBurger
Many “Whelps”! Handle It!
Thinking of the gigantic powershell script I wrote to go through our hundreds of mailbox databases across 16 mailbox servers to determine how fragmented they were, then defragmenting them and reclaiming whitespace, all while having to worry about free space on the iSCSI LUN's provisioned from our SAN ..... I don't miss that monthly maintenance Sunday mornings. Or how you have to apply rollup updates to each server one at a time and it's a whole process, not just running a patch file and being done with it.
Yea. Exchange Online is the way to go.
Last edited:
Themacabre
Member
As much as I agree in general, there are many clients where that isn't possible due to a variety of factors, and some where it was only possible very recently (a year or two ago). Ideally though, yes absolutely.
Northeastmonk
Gold Member
I ran into CrowdStrike recently and it blocked me from performing an assessment. Can’t say I see it very often, but it sure does its job. I can also see people moving away from it. Piss off the non technical people, they won’t do business with you.
StreetsofBeige
Gold Member
Our company lucky. Nothing affected me (unless that temp issue I had above was part of it.... it fixed). A memo came out saying if you work in certain programs you might get affected. I dont use those, so I was good.
eddie4
Genuinely Generous
It's hilarious. Southwest still uses Windows 3.1/95 on most of their systems and they were not affected. 
www.govtech.com
Why isn’t Southwest affected by the CrowdStrike/Microsoft outage?
Answer: Because it’s using an outdated version of Windows.
www.govtech.com
StreetsofBeige
Gold Member
If it aint broke, dont fix it.It's hilarious. Southwest still uses Windows 3.1/95 on most of their systems and they were not affected.
Why isn’t Southwest affected by the CrowdStrike/Microsoft outage?
Answer: Because it’s using an outdated version of Windows.
My buddy who worked for decades at space equipment companies said that the shit theyd make for the space shuttle works on crude DOS or UNIX kind of programming. No windows. This was way back, so maybe things changed. But at the time he said that the instructions needed are basic and dont need the crash prone and bloated Windows stuff. So it was crude. My first company I worked at out of school, we used that ugly ass green screen Unix looking shit for inventory control. To go from screen to screen you had to know or someone told you to go to P30 or P22 for this screen or that screen. It would be like P15 is pricing screen, and P28 was to enter forecasted units etc.... There was no logic to it.
But from what I remember it worked. I dont remember ever needing to reboot the program or PC.
Last edited:
Nitty_Grimes
Made a crappy phPBB forum once ... once.
I'm sure the NHS is still on XP - unless my consultant just likes to make his computer look like XP - which is highly unlikely.
Laptop1991
Member
you can't blame companies for using older software when the newer ones cause this type of chaos, i mean everyone panicked over the Y2k bug that was going to shut everything down and nowt happened lol, too much control now and not enough options. because money!.
Last edited:
BlackTron
Member
Anyone could have foreseen that as a bad combination. Now that it has been demonstrated IRL, will anything change?
No.
The Stig
Banned
I'm an IT professional of nearly 17 years. I just moved from USA back to the UK last week and am having a few weeks off before getting back into the workforce.
I cannot believe how good my timing is/was.
If I waited one more week to move I'd be fucked. BULLET DODGED
If I waited a couple more weeks and continued working at my old position I'd be fucked (theyre having a terrible time right now). BULLET DODGED
Just call me NEO or Max Payne lol
My old boss was always VEHEMENTLY against crowdstrike. He is feeling quite smug right now, but sadly many of our clients at my old job used it anyway.
I cannot believe how good my timing is/was.
If I waited one more week to move I'd be fucked. BULLET DODGED
If I waited a couple more weeks and continued working at my old position I'd be fucked (theyre having a terrible time right now). BULLET DODGED
Just call me NEO or Max Payne lol
My old boss was always VEHEMENTLY against crowdstrike. He is feeling quite smug right now, but sadly many of our clients at my old job used it anyway.
Bitmap Frogs
Mr. Community
Yeah, on some of our products we do exactly this.
Zathalus
Member
There is no alternative. You can't go without some sort of antivirus or randomsomware protection and in order to stay ahead of security threats those same protection programs need to be updated often. The issue is simply that Cloudstrike did not follow best practices and is now paying the price.
BlackTron
Member
There is no alternative to a cloud based solution? Updating it from the cloud isn't enough?
Hey I assume it's a bit deeper making sure it works than grabbing the latest ccleaner but still, there's no alternative?
Zathalus
Member
It is updated and managed from the cloud, the problem was the local application crashing computers. It’s simply a faulty update that was pushed to clients. It’s not a threat protection running remotely.
BlackTron
Member
Well I have to concede you're gonna need to have updates for vital protection be mandatory. Just make sure the update doesn't break everything. Damn.
Saber
Member
It's hilarious. Southwest still uses Windows 3.1/95 on most of their systems and they were not affected.
Why isn’t Southwest affected by the CrowdStrike/Microsoft outage?
Answer: Because it’s using an outdated version of Windows.
Most of Brazil computers also don't uses windows 11, some still use XP and 7.
I think all this situation should make people rethink this approach.
BadBurger
Many “Whelps”! Handle It!
My manager was the same way. We absorbed another large org a few years ago and they were using CrowdStrike solutions. My boss fought hard (along with a few other IT division managers to their credit, such as the database mgmt team) to migrate them to our solutions ASAP and to avoid CrowdStrike at all costs in the future.
He had such an air of "I told you so" the morning after this outage during meetings. I found myself performing the approving Miyagi head nod a few times.
Hudo
Member
Will any company affected by the outage sue CrowdStrike? Most likely not.
CrowdStrike will just do the same thing other cloud bullshitters do. Gaslight people: "it was a hardware/software problem. But we fixed it! We're awesome at support. Glad that you're paying us, right? And not the other guys."
So yeah, shit will only get worse.
nikolino840
Member
Unknown?
Member
You shouldn't have all of your money in a bank(virtualized) anyway.
There is, don't have essential functions based on software. If you can't sell product without software, you're doing it wrong. These companies that couldn't make transactions with cash are moronic!
Zathalus
Member
Not based on software? How do you run an international company and be competitive without it? Absolutely infeasible.
ForAcademicPurposes
Member
I agree.
And to to the rest of the goombas around the world. Ya'll need to rethink this and start investing in what is tangible.
Unknown?
Member
Sorry, I didn't mean not having it all together, I should have said not having backups for this kinda situation where you can still do business without it is moronic.
poppabk
Cheeks Spread for Digital Only Future
When my job sent out a text using the emergency number saying to restart your computer multiple times and then a link I thought that it was a phishing attack of some sort.
BlackTron
Member
Yeah that would have been my reaction, at least until I checked GAF.
Zathalus
Member
Oh yes, that makes sense. Redundancy on systems like this can be critical.
West Texas CEO
GAF's Nicest Lunch Thief and Nosiest Dildo Archeologist
Let's cool off from the politics, guys.
NecrosaroIII
Ultimate DQ Fan
Today has sucked. I work in logistics. There is still sort of a ripple effect happening from this. Lots of cargo is backed up and delayed or missing. All of my clients are pissed and taking it out on me.