What Caused Such a Widespread Tech Meltdown?
There have been two issues affecting Microsoft systems in quick succession, adding to the confusion over a global outage.
An outage has hit computers around the world, affecting airlines, hospitals, retailers and other businesses. At the heart of it, there have been two issues in quick succession involving widely used Microsoft systems, and some users may have been affected by both of them.
On Thursday, some Microsoft clients in the central United States, including some airlines, were affected by an outage on its cloud service system, Azure. Microsoft’s cloud service status page indicated the company had identified a preliminary cause.
Some users may still be unable to access certain Microsoft 365 apps and services, including Teams video conferencing. The company was aware of the issue “affecting a subset of customers,” a Microsoft representative said in a statement. “We acknowledge the impact this can have on customers, and we are working to restore services for those still experiencing disruptions as quickly as possible.”
Separately, on Friday many Windows devices experienced problems involving CrowdStrike, according to the Azure status page. “We’re aware of an issue affecting Windows devices due to an update from a third-party software platform,” according to the representative. “We anticipate a resolution is forthcoming.”
That outage was caused by a flawed security update by CrowdStrike. George Kurtz, the company’s chief executive, said in a statement that it was “actively working with customers impacted by a defect found in a single content update for Windows hosts.” He added: “This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed.”
The issue appeared to be with an update to CrowdStrike software called Falcon Sensor, according to Lukasz Olejnik, an independent cybersecurity researcher and consultant. An updated fix of the software has been sent to computers, but Mr. Olejnik said outages would probably persist because it was not clear how to fix the huge numbers of computers that had already been affected.
A big part of the problem lies with the current suggested solution, which is to reboot each computer manually into safe mode, delete a specific file, and then restart the computer normally. Security experts said that while it is a relatively simple process, there is no way to automate it at scale.
“There is a workaround, but it requires manually tampering with Windows systems files in recovery mode,” Mr. Olejnik said. “Such practice is in general not advised ordinarily, as mistakes may cause other problems.”
That leaves affected organizations with a major quandary: how to raise the huge numbers of qualified professionals to go through and update the computers one by one. It could mean that even with the problem known and a patch sent, systems outages could persist for much of the day, if not longer, according to experts. Those with organized and well-staffed information technology teams could potentially fix it more quickly, said Mr. Olejnik.
Although it may seem counterintuitive that a single bad update could have such devastating consequences, a major problem was that the software being updated was doing critical cybersecurity tasks. CrowdStrike’s Falcon Sensor software, at the center of the outages, scans computers to look for viruses and other malicious attacks.
“One of the tricky parts of security software is it needs to have absolute privileges over your entire computer in order to do its job,” said Thomas Parenty, a cybersecurity consultant and former National Security Agency analyst.
“So if there’s something wrong with it, the consequences are vastly greater than if your spreadsheet doesn’t work.”
It is not clear whether the initial Microsoft Azure outage was coincidental or linked to the crashes caused by the CrowdStrike update, but security experts said it was possible. Most likely, if the two are related, it would be because CrowdStrike’s update affected computers responsible for running Azure, triggering its shutdown.
“If I were trying to think of a causal link between the two, it could be that the systems required for the connection to Azure were initially hit by the particular CrowdStrike problem, making the service not available,” said Mr. Parenty.
The outages, which shut down airlines, hospitals and emergency response services across the globe, indicate how delicate technology systems can be when they are so interdependent.
“This is a very, very uncomfortable illustration of the fragility of the world’s core internet infrastructure,” said Ciaran Martin, the former Head of Britain’s National Cyber Security Center who is now a professor at Oxford University’s Blavatnik School of Government.
It also underscores an uncomfortable reality that software companies face few liabilities for major outages and cybersecurity incidents. The economic and legal penalties for such massive outages can be so minimal that companies are not motivated to make more fundamental changes.
“Until software companies have to pay a price for faulty products, we will be no safer tomorrow than we are today,” he said.
Commentaires