Information To Protect What You Grow

Businesses Lose Millions In Revenue After Backup Systems Fail

Written by Jeffrey Forbes | Nov 16, 2017 2:00:00 PM

"When people consider cybersecurity, too often they think of high-level data encryption, secure sockets layers (SSL), and high powered firewalls and antivirus protection", says Charlie E. Bernier Principal Consultant and Cyber Insurance Expert at ECBM.  "Yet overlooking simple steps to protect a company’s information technology resources can prove just as costly."  Thinking through issues like adequate backup systems, employee training, and network setups can save a company millions of dollars.  A series of outages at major airlines over the past year have highlighted the specific importance of planning backup systems properly.  Delta Airlines, United Airlines, and Southwest Airlines have all suffered major information technology outages that canceled hundreds or thousands of flights cost these companies millions of dollars in revenue.

Issues Caused By Hardware Malfunctions

In July of 2016, Southwest had a major computer system failure.  The failure took almost an entire day to fix and required the company to reboot the entirety of its computer systems.  The failure canceled over two thousand flights and delayed many others.  The failure also prevented customers from booking flights online.  Industry experts estimate the total losses to Southwest from the system failure will reach fifty million dollars.  A single faulty router caused the system failure and the $50 million in losses, as Southwest’s backup systems were not designed to handle this problem and failed when the router failed.  The router failure caused a chokepoint in the company’s data center, and because the router only suffered a partial failure, the backup systems did not kick in the way they were supposed to.

More Lost Revenue For Other Airlines Due To Computer System Crash

One month later, Delta Airlines suffered even bigger losses due to a similar failure to plan through their backup systems.  Delta’s systems failed on a Monday morning.  The Delta failure also canceled more than two thousand flights with significant further delays.  It also took Delta approximately three days to fix.  Estimates for lost revenue due to the incident place the figure at over one hundred and fifty million dollars.  A single piece of technology also caused Delta’s system failure.  In this case, the culprit was a power control module.  When the power control module failed, it caused an electrical surge that blew a transformer.  With power lost, the airline’s computer systems should have switched over to backup systems.  However, the company had failed to connect approximately 5% of their servers to backup power.  This caused massive system-wide instability as some systems came back on and others failed entirely. 

Your Disaster Plan Should Include A Plan For Hardware Issues

The sheer size of these losses highlights the importance of adequate backup systems.  Pieces of technology will fail.  Natural disasters will occur.  Human beings will make mistakes.  Building double and triple redundancies into your system can keep business operating as normal even when these events happen and avoid significant economic losses as well as damage to your reputation as a company.    Charlie advises businesses to "make sure you are thinking through not just the possibility that each device might fail, but how they might fail and what the knock-on effects are when building your company’s systems."