• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.

Worldwide Microsoft Outage: Flights Grounded, Sky News Off Air, Workplace Systems Down

AV

AV

We ain't outta here in ten minutes, we won't need no rocket to fly through space
Developing story:

www.telegraph.co.uk

Microsoft outage latest: Chaos as internet down and flights grounded around the world - latest updates

Television channels, airports and banks around the world have been knocked offline in a massive outage causing Windows computers to suddenly shut down.
www.telegraph.co.uk www.telegraph.co.uk

From Azure via a colleague:

"A backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks."

IT go boom. Could be a short work day today for a lot of people.
 
BadBurger

BadBurger

Many “Whelps”! Handle It!
Our DR assets in US Central went down 8 hours ago. They're still down.

"Current Status: We have determined that a backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks. We have confirmed mitigation for Azure Storage and Compute resources. We are continuing to investigate and mitigate a number of downstream impacted services that have not returned to a fully healthy state. "

There's also a second from 6 hours ago:

"Impact Statement: Starting at 22:12 UTC on 18 Jul 2024, you were identified as a customer using App Service in Central US who may experience issues while performing service management operations such as site create, delete and move resources on App Service (Web, Mobile and API Apps) applications. Autoscaling and loading site metrics may also be impacted."

I suppose none of these affected companies had any kind of DR. Oops.

Edit: BTW, I am pretty sure Microsoft made it so that anyone, even if you don't have a sub, can view the Service Health dashboard in the Azure portal. I am already logged into Entra (Azure Active Directory) so I cannot confirm. But try browsing to portal.azure.com, and if you get in, do a search for "service health"

Edit 2: Looks like Crowdstrike is headquartered in Texas, near the Central US datacenter (well, near-ish). I wonder

S7hVorH.png
 
Last edited:
Power Pro

Power Pro

Member
Now I think I realize why my overnight package landed in Kentucky when I'm no where near Kentucky...probably gonna be delayed now cuz of this crap.
 
BadBurger

BadBurger

Many “Whelps”! Handle It!
Yea, shares are down with Microsoft 3% as well. Heads were already going to roll, now that their money is being messed with there could be a VP or C-suite head included in the bunch as well.
 
Lord Panda

Lord Panda

The Sea is Always Right
FunkMiller said:
Be really nice if this slammed the brakes a bit on the 'cloud revolution' and software as a service. People might realise that having too much stuff that isn't local is potentially very risky.
Click to expand...

It's an issue of sovereignty with putting all your eggs into one (or small number) of baskets outside of your control.
 
Last edited:
Wildebeest

Wildebeest

Member
Both closed source and open source software is becoming so complex and reliant on multiple software from different sources regularly updating. It is a huge challenge.
 
The Stig

The Stig

Banned
lol crowdstrike

I remember my company's head of security arguing with some of our customers to NOT use crowdstrike a few months ago. can't remember why but I bet he is feeling quite smart right now
 
Power Pro

Power Pro

Member
When you're being used in millions of computers around the world, this shit should be properly tested on hundreds, if not thousands of computers before it gets put out on a massive scale like this. This stupid companies need to be accountable for this kind of crap. They're probably causing a lot of businesses huge problems today.
 
Rossco EZ

Rossco EZ

Member
Don’t wanna sound stupid but will this affect my home pc at all? Been hearing on the news that it’s making pcs blue screen or something but is that a windows update that’s done that?
 
BadBurger

BadBurger

Many “Whelps”! Handle It!
At least the fix looks quick on an individual basis. I feel sorry for desktop and deployment management teams out there that are going to have to deal with thousands of systems.
 
darth.shrimp

darth.shrimp

Member
Jayjayhd34 said:
Private Hospital Im in is having issues.
Click to expand...
I hope it's not too severe. I work at a pharmacy, and while we're not personally affected, all of our suppliers are, so we can't place any orders for medicine (or when we can, we're not even sure the supplier can do anything about it).
 
Hudo

Hudo

Member
All hail cloud computing. It will truly solve everything.

Edit: It seems to be even more embarrassing than cloud stuff: It's some local anti-virus software. Suddenly I have early 2000s flashbacks.

Edit 2: So I went onto the CrowdStrike website because i have never heard of this thing before (I apparently am lucky to never have worked anywhere where they really had absolute retards in charge of IT security, it seems).
And it seems like the worst thing I have ever heard: Protection of end devices via some cloud service. This sounds like one of the worst ideas anyone could ever come up with.
 
Last edited:
DavidGzz

DavidGzz

Gold Member
I was screwed last night here at my workplace. 911 CAD system was down most of the night. Only 1 out of our 7 work stations were operational. Co-workers had to use pen and paper to document calls. We were back in the 70s. Windows 11 seems to be fine? Idk, all the Windows 10 pcs were borked and are still restarting over and over.
 
Last edited:
BadBurger

BadBurger

Many “Whelps”! Handle It!
Punished Miku said:
I wanted to get some shit done today lol. I might have to come in this weekend now ...
Click to expand...

There's actually an easy fix if you want to try it. From some Forbes article - do so at your own risk. Another article states that CrowdStrike is currently deploying a fix:

There is a workaround, he added.
1. Boot Windows into Safe Mode or WRE.
2. Go to C:\Windows\System32\drivers\CrowdStrike
3. Locate and delete file matching "C-00000291*.sys"
4. Boot normally.
 
Last edited:
You must log in or register to reply here.
Top Bottom