August 28, 2024 will go down as a particularly bad day for George Kurtz – CEO of ‘Crowdstrike’ – a tech security firm responsible for millions of ‘blue screens of death’ across the world, as one of his company’s updates to the Microsoft (MS) Windows operating system achieved precisely what it is designed to prevent.
Ironically, the resulting worldwide systems failure had nothing to do with hackers or even an automated penetration testing exercise gone wrong. Quite simply, Crowdstrike’s product caused mayhem as one of its software updates glitched and knocked out Windows and other MS-driven systems across the planet. Millions upon millions of MS powered computers just outright failed to function.
Banks, hospitals, shops, utilities companies, emergency services and many other public systems simply stopped working. Employees, managers and domestic PC users across the planet threw up their hands and waited for a fix.
But it turns out that due to the catastrophic nature of the update, many of the world’s computers now have to be repaired and rebooted manually. Some people could be offline or working at very reduced capacity for a long time until they can find enough information technology (IT) support to get things fully reinstated.
As the horrendous extent of the disaster became clear, an MS spokesperson stated during the day that the outage was caused by a defective update to its ‘Falcon’ security software for Windows hosts. Falcon’s coding under the hood is evidently written by MS subcontractor, Crowdstrike.
Cost of the crash
And it wasn’t only the world’s computers that crashed. So did Crowdstrike’s share price. And how.
Unsurprisingly, the financial markets didn’t treat the organization’s prospects with much sympathy, as an estimated 20% was initially wiped off the company’s trading value. Figures later rallied by around 11% – but the firm ended their fiscal day about 9% under where they had been before the debacle.
It’s a fair guess to assume that someone at Crowdstrike got royally Fired with a capital F!
Implications for risk management
What’s slightly more concerning about this incident is that it highlights the reliance upon monoculture systems across our planet. As both in agriculture, animal husbandry and IT – it’s very bad for any ecosystem to rely upon a single pillar holding up everything; when it fails, it fails spectacularly.
Unfortunately, monocultures grow naturally because they leverage economies of scale. In simple terms, it’s cheaper to manufacture and distribute a single solution than product or service providers offering multiple options to their customers.
Likewise, a farmer makes more profit and increases efficiency if they stick to one crop and dedicate all production efforts to it on one farm or several. Interbreeding of animals and even humans can produce disastrous results. You only have to visit certain isolated rural communities across the world to see how well that pans out.
Cheapest isn’t best.
Examples of cheap stuff beating better quality products due to mass market adoption abound.
Back in the 1980s the Betamax video tape cassette player was seen by many people as a superior product design to the VHS machine – but VHS became the worldwide de-facto standard simply because it was cheaper. More people bought it than they did Betamax and after a short time video cassettes (at least for showing mainstream movies) were only produced in VHS format.
Likewise, Apple computer fans the world over, who might currently be enjoying some smirking schadenfreude after Crowdstrike’s cock-up, will tell anyone who will listen (or won’t!) that Macs are miles better than Microsoft-powered PCs. They say that they’re more secure, have better hardware, and are much more aesthetically beautiful blah blah blah…
But Macs are often three or four times the price of basic Windows machines, so the monoculture was created whereby 80% or so of the world uses Microsoft’s arguably inferior operating system to macOS.
These disasters are nothing new.
Historical examples of monoculture failures might include the Irish potato blight of the mid 1840s, or the worldwide panic, fortunately unfounded, which was caused, again by Microsoft – by their ‘Millennium Bug’ scare (Y2K) in the months running up to December 31st, 1999. Everyone thought their computers would be rendered forever unusable. Many people canceled flights and hospital appointments, fearing the worst.
In the case of the potato famine, many thousands of Irish people immigrated to the USA’s Eastern Seaboard. The Y2K bug firmly cemented Bill Gates, Microsoft’s then CEO, as ‘The Prince of Darkness’; an unfortunate nomenclature that stuck to him for many years.
In short, if everyone uses the same software, we’re all vulnerable when something goes wrong such as a worldwide data breach from one provider. From whole operating systems like Linux, macOS and Windows, down to a simple package such as website optimization software Orca SEO; these things normally operate perfectly well – but if everyone’s using the same stuff it’s ‘all she wrote’ when disaster strikes.
In the case of SEO plugins it might mean that your home page link could fall off P1 of Google – if you’re a heart surgeon relying on the AI of a robotic arm – your patient may easily die.
In the mood for change?
One positive takeaway from all this might be the realization amongst corporate computer buyers that moving away from just one OS for an entire organization might be a wise precaution. Large conglomerates that can afford to do so might be wise to keep, say, an on-premise Linux backup to their systems if their primary cloud-based MS Windows systems fail.
It also makes you wonder how World War III might be fought – not with tanks, troops and nuclear warheads, but a relatively simple virtual spanner thrown into the cogs of an entire nation’s IT systems by aggressive state-sponsored hackers.
Indeed, in summary, Raj Joshi, SVP for Moody’s Ratings, summed up the situation well. He stated that there were no real winners or losers in the Crowdstrike disaster, but highlighted the world’s dangerous over-reliance on monocultures in IT. He said:
“This incident calls into question CrowdStrike’s software engineering practices… it also underscores growing vulnerabilities in global cloud infrastructure from increasing points of failure.”
Perhaps the world will start to wake up to the dangers of monocultures, especially in IT systems, as one day their failure could threaten our very survival as a species on Earth.
