Everything, Everywhere, All at Once: The Key to IT Crisis Management Effective Incident Response

September 28, 2023

To maintain CyberOps and ITOps resilience, a game plan is good. But muscle memory is better.

As any professional athlete will tell you, resilience is key to success and the only way to attain it is to train, train and train some more. Such is the case for incident response (IR) teams which have to remember their game plan and have everything, everywhere working in harmony. That’s the way to weather the storm and recover as quickly as possible.

Uphill battle

Today’s IT ops and security teams too often find themselves in an uphill battle. Attackers may not always carry the element of surprise or unleash sophisticated attacks, but their persistence and stealth can be unnerving. They are relentless, ever ready to exploit a vulnerability, use a stolen credential, or take advantage of a slow reaction or lapse in focus. Plus, they have a thriving underground economy offering all the tools, services, and knowledge they need to launch attacks. One security vendor saw cyber threats jump tenfold in just the first six months of this year. Yet many more sneak in under the radar.

On the defensive end, network protectors face a relentless tide of challenges. There’s a persistent and chronic skills gap, with an estimated shortfall of cybersecurity professionals globally in excess of three million, including more than 410,000 in the U.S. And thanks to pandemic-era investments in digital infrastructure and a shift to hybrid work, corporate attack surfaces have never been more expansive. Two-fifths of global organizations think these are “spiraling out of control.”

Against this backdrop, breaches and security incidents are inevitable. And they’re having a critical impact on enterprises. The average cost of a data breach globally is $4.35M, and in the U.S., it is $9.44M.

The speed of incident response is also a top security operations (SecOps) challenge. Why? Because too many organizations are laboring with multiple tools and siloed data, which give conflicted signals that are difficult for SecOps teams to prioritize. The data they do get to work with is often stale, incomplete, and riddled with false positives. Systems management solutions are unable to provide the rapid visibility analysts need to detect and contain emerging threats.

Practice, practice, practice

So how can organizations get back on track? In short, careful planning and repeated practice. First, decide what your corporate crown jewels are. It may be a customer transaction database, or some closely guarded intellectual property. What can the organization not live without? Then it’s about war gaming to find where the gaps in security posture are that need filling: an unpatched server here, a misconfigured access point there. Know what you look like to an attacker. Use red teams to simulate real-world attacks covering an exhaustive range of scenarios. This is about building resilience to minimize the chance of a successful breach. An ounce of prevention is worth a ton of cure.

But, as discussed, prevention can only get you so far. That’s why it’s critical to test those incident response plans. This is the opportunity to get everything and everyone working together, all at once. Get your critical IR stakeholders in a room twice a year to run and test those foundational processes around change management, problem management, communication structure, disaster recovery plans, business continuity and more. This is the only way you’re going to find those single points of failure that won’t appear during normal operations.

The key here is to keep those operations running as smoothly as possible even during an incident. It’s about resisting the human urge to panic and try something new when a crisis hits. You want to get to a stage where when a crisis comes along it’s not a crisis at all but just second nature for everyone involved. Everything, everywhere whirrs into action because you planned and trained for this. In short, you know what to do when you get hit in the mouth.

When seconds count

Emergency responders do this. When lives are on the line, they need to prepare for every eventuality. Military units do it too. And of course, professional athletes practice their critical plays until it comes as naturally as breathing. But analogies aside, what do incident response teams actually need to maintain business operations, recover rapidly, and minimize the damage to the organization?

First, they require real-time visibility into the entire IT environment. IR teams need granular, accurate, and rich data to scope an incident, then investigate efficiently and quickly. And they need instant control, to take corrective action when necessary. That means working from a single source of truth – a platform that can scale to hundreds of thousands of endpoints without breaking a sweat. That kind of visibility and control won’t just allow IR teams to fix any issues unearthed in war-gaming tests, it will also enable real-time threat hunting, as well as incident investigation and rapid remediation, if a worst-case scenario becomes a reality.

When seconds count, speed and precision are critical. But so too are cool heads and well-trained teams.