Tales from Trenches: What’s on Your Shelf?

Don’t you hate it when you forget something? Like when you take your kids on a hike and forget a snack, or take your dog for a walk and forget those-oh-so-necessary little bags? It results in fuming kids and a very serious dog-culture no-no, not to mention guilt. Insert your similar stories here, but in those two scenarios, the treats and the those-oh-so-necessary little bags could represent technology in the people, process and technology triad of despair. You could forget the people/dog or process, and you may have other types of disasters equating to the same outcome – failed execution. In this blog entry, I’ll share two real-life IT security scenarios where I have witnessed failed execution, and why.

Scenario 1: About That Technology On The Shelf (The One You Forgot You Had)

Earlier in my career, I worked as a contractor/ advisor to the CIO of an organization with a critical mission (names withheld to protect the innocent). There I had the unique opportunity of observing the IT and IT security functions and making recommendations based on my review and analysis of key aspects of those programs. My framework was ISO 17799 (yes – way back then). My examination uncovered distributed IT and IT Security functions with islands of excellence and issues scattered geographically. One of the challenges I decided to make a priority was logging and incident management. As I have seen throughout my career, systems were logging locally and set to write over logs to avoid filling up storage and taking systems offline – bad in the incident management and response circles (I can see heads nodding in agreement). After interviewing multiple Subject Matter Experts (SMEs), I learned that a very well known Security Information and Event Management (SIEM) system had been purchased. After more digging, I found that over $1M had been spent in hardware and software, but the technology was never deployed. What happened? I poked around more and found that several factors led to the failure of this “project” (using quotes because “project” usually includes some type of management). There were the usual suspects, like a lack of leadership support (a topic for another blog) and cultural issues (another focus for another blog). Some individuals told me the primary culprit was lack of a process for implementing a large-scale enterprise project. Others said it was a lack of people to manage a project of that magnitude and a thought-out plan for the on-going operational work required for a SIEM.  With no approach or strategy and no people to operate this system, team managers were pushing the notion of doing something different away – hard! It is hard to believe that managers in an IT organization have that kind of power – but I still see this to this day. Without processes such as strategy planning, change management, program/resource management, the list goes on, we have nothing to base the way we are supposed to do things on, and the result is a mutiny. So the technology sat on the shelf? What’s on yours?

Scenario 2: More Technology. No Process. No People. 

Fast forward six years, a change in my role brought increased responsibility. I was now leading the security organization for a global company. I’d developed my roadmap and strategy, and I was executing. I had initiated a discovery project to review the state of logging and monitoring, and we were just getting started when one of my project managers came to me. This person had been with the company for well over ten years, and because of the continuity and history he possessed, I listened carefully. The story sounded just like the scenario I mentioned earlier. It was yet another failed SIEM project, and it was with the same vendor! This situation had all the same markers – lack of leadership support (there was a new sheriff in town - me) and cultural issues (I was already working to knock down barriers between IT and IT security). There was no formal PMO and no roadmap or strategy before my arrival and when this project initiated and subsequently failed. I was working with other IT executives to ensure we made plans and decisions that were well thought out and prioritized based on an agreed approach. I started to try and pull the pieces together. I thought - maybe we can get this project back on track. I found that there was an incredible investment made here – over $1M. When the implementation failed, a decision had been made before my arrival to stop paying maintenance on the solution. If I was to consider restarting the SIEM project, I had to take into account the payment of back-maintenance plus new maintenance and implementation costs. In addition, all the hardware had been re-purposed (cannibalized!) for other IT-related initiatives. Rather than derail my current discovery project for centralized logging and monitoring, I decided to follow the PROCESS and see what the recommendation would be at the end. The output was collaborative and complete (PROCESS/PEOPLE). It included all the history and unique cases related to the organization to avoid “gotchas” when it was execution time (TECHNOLOGY). The discovery project output also included resource planning for implementation and on-going operations (PEOPLE). We ditched the notion of breathing life into the old failed project and saved money and time by going with a different approach – MSSP (Managed Services Security Provider). We kept in mind that just because some of the work was about to be outsourced, it didn’t mean that the MSSP operated independently – we needed to manage the vendor and ensure a process existed between us for incident hand-off.

In the illustration above I can relate to the outcome of “shelf-ware” to Scenario 1 very well. Moreover, the fix wasn’t to force the SIEM into implementation and enterprise-wide adoption by hammering IT staff into submission. The fix was an IT process review, and security strategy assessment with careful thought put into resources and solving the problem. In Scenario 2 the successful outcome was based on balancing People, Process and Technology. This is a geek version of “Goofus and Galant.”

Benjamin Franklin once said, “An ounce of prevention is worth a pound of cure.” Striking balance with People, Process and Technology will ensure prevention is achievable and outcomes are successful. So take a look on your shelf. You might have an implementable solution right there – if you get some help with integration processes.