The inevitable has happened—you’ve been hacked. Or maybe your CEO lost his laptop at the airport, critical equipment got fried when someone shut off the A/C in the server room, or an intern deleted your entire SQL database. The decisions you make in the next few hours will be critical in determining the depth of impact on your business for months or years to come.
It would be nice to think that nothing like this will ever happen to your organization, but the truth is, it’s not a matter of if an incident will occur, but when it will occur. Hopefully it happens after you’ve developed your Incident Response Plan—not before. If you already have one, great! Use this post to double check your plan and make sure you haven’t forgotten anything. If you don’t—you’re taking the same chance the designers of the Titanic took when they chose to skimp on the lifeboats.
The rate of enterprise data breaches and their financial impact have been increasing steadily and show no signs of slowing. Currently, each individual record lost or stolen causes over $200 worth of damage to the organization, on average. It doesn’t matter if you were intentionally attacked by an outside hacker or a disgruntled employee, or if the loss was caused by an honest mistake. Your organization stands to lose production time, revenue, the trust of customers and partners, your brand image, and even the business itself. Living in denial about the likelihood of an emergency is not an option, so what can you do?
First things first—most of the time, incidents can be quickly mitigated or avoided entirely by proper education and best practice compliance. 90% of data breaches are easily preventable. Teach your employees how to easily identify fake emails to stop phishing attacks. Explain to your finance department why “password” is not an acceptable password. Ensure that database backups are performed in a timely manner (and stored securely offsite). Start simple with your team, but build their technical expertise over time and empower every single member to take ownership of their role in safeguarding the interests of the organization. It’s true what they say: “An ounce of prevention is worth a pound of heartburn medication.”
You should be conducting ongoing training and performing compliance audits with your entire staff regularly. You should. But either way, you must develop an Incident Response Plan for when your best efforts are overcome, because someday they will be. If you have an Incident Response Plan in place, you can stay focused through the chaos, and you stand a fighting chance of preventing permanent, crippling damage. Like the Boy Scouts, be prepared. A good Incident Response Plan addresses policy, strategy, communication, documentation, areas of responsibility, access control, tools, and training.
Policy
The first goal of any Incident Response Plan should be prevention. Establish policies that determine how employees and others can use the network, and lay out which methods will be employed to monitor and enforce compliance, and penalize infractions in a clear, concise manner. Present employees with these policies during onboarding, annually and upon revision, and make them readily accessible to all. Consider implementing a landing page for those connecting to your wireless network that requires acceptance of the Terms of Use. If you don’t introduce policy, you can’t expect users to observe company guidelines and legally speaking, you won’t have much recourse if their actions jeopardize the network. Additionally, you should investigate Cyber Liability insurance, to see if it makes sense for your business.
Strategy
This is the heart of your Incident Response Plan. When an incident occurs, these are the specific steps your Incident Response Team will follow to identify, contain, eradicate, and recover from the problem. As this may be one of the most impactful procedures your company will ever develop, seek input from all relevant sources—the IT department, ownership, attorneys, insurance agents, even individual department heads. Something important to remember is that not all incidents are created equal, and as such, do not require the same level of response. Designing a tiered system based on severity of impact will help determine the amount of resources, equipment, and effort to assign in any given situation. The strategy portion of your plan must be meticulously planned, and will take a long time to get right, but it’s worth it. Once you have your strategy, it will need to be re-written with clarity and conciseness in mind. It must be accessible and easy to read, so that your team will have no difficulty understanding and following the plan when the time comes.
1) Identification
Before your team can fix any issue, it must be confirmed that an incident has indeed occurred. They may be notified directly by an end user, get an alert from a security appliance, or notice it themselves, and they will begin to examine log files, error messages, intrusion detection systems, firewall rule sets and the like. As soon as a breach is suspected, documentation must begin immediately, even if that initially means using a smart phone to record audio and video during the first few hectic moments. Pains must also be taken to avoid destroying any evidence.
2) Containment
The goal of the containment phase is to limit the extent and spread of the damage as much as possible. In the case of the CEO’s lost laptop, perhaps a remote wipe is all that’s needed. If you suspect that your network has been infiltrated by hackers, you may need to quarantine the apparent point of penetration and immediately begin assessing the adjacent systems. This could escalate to disconnecting major portions of your network, or even killing all access to the external web. Once initial quarantine has been completed, and you’re certain the problem can’t spread any further, you will also need to contain—or perhaps maintain—original verbatim copies of the infected systems for evidence and future analysis. Perform system backups creating forensic quality images on non-network connected devices, and as always, continue detailed documentation throughout the process. The final consideration in the containment phase is one of continued isolation. Ensure that all infected systems stay separate from the network during the following two stages.
3) Eradication
Eliminate the threat and attempt to solve the problem, whatever it may be. If the intern deleted your database, there’s nothing left to “eradicate” per se—he took care of that for you, but you can still try to repair the damage. Hopefully proper backup policies were followed and you can restore most of the lost data. Of course, in the case of malware or a security breach, we do want to eradicate that. After you’re sure you’ve acquired all possible forensic evidence in the previous step wipe the affected systems and complete full system restores. Replace damaged physical equipment, if any. Take this opportunity to harden the affected systems and the area of the network that was exploited. Close unused ports, disable nonessential services, patch the exploited vulnerability, and then rescan everything to ensure the infection is truly eradicated. As always, document all progress as you go.
4) Recovery
Schedule a date and time that will have the least impact on the production network and then reintroduce the repaired systems to the environment, install any newly replaced hardware, and perform rigorous scans once everything is back in place. If your client database was breached, quarantined, remediated and then restored to service, it’s still certainly possible that a resourceful hacker managed to hide a dormant infection elsewhere in the network that will go right back to work on your client database as soon as it’s back up and running. You’ll also need to determine the appropriate length of time the affected systems remain under continued heightened monitoring to be sure this is not the case.
Communication
Your Incident Response plan should also detail a list of who to notify, along with a variety of contact methods based on severity and area of impact. This will include members inside and outside of the organization. Any real emergency will necessitate alerting the on-call member of the response team and likely, the head of the affected department, but not all incidents require you to call the CEO, police, fire department and National Guard. Determine in advance when it is appropriate to notify ownership, legal counsel, law enforcement and your insurance company, and define the process by which employees, partnered organizations, and the general public need to be informed.
Documentation
This is an important component of the plan that too often does not get the attention it deserves. Hopefully, your organization will never be subjected to an event so serious that a legal trial is necessary, but if it ever does, sloppy work in this department will be serious cause for regret. Even if you manage to avoid a situation like that, proper documentation can be instrumental in mitigating future risks and be an excellent referral source of information in the case of repeat incidents. This part of the plan should codify how documentation is recorded, whether that be electronically or physically and the proper chain of custody to prevent evidence corruption. Decide how much of the “who, what, when, where, why, and how” is to be recorded and in how much detail, and develop incident response checklists to facilitate collection.
Areas of Responsibility and Access Control
Define the individual roles of the Incident Response Team, their separate areas of responsibility, and designate members to fill those roles. Keep this list updated with appropriate names and contact info at all times. When an emergency arises, this eliminates the team scrambling to figure out who is in charge. Knowing one’s role in advance during a stressful situation can help to avoid panic, limit in-fighting and blame-laying, and ensure forward momentum. Your team is not just made up of members of the IT department. Don’t forget to designate representatives from each production department in the organization, Human Relations, Public Relations, Legal, Finance, and be sure that all members have pre-authorized access to all systems and resources necessary to fulfill their role. As a tenant of good policy, if a member does not ordinarily need those resources during the normal course of their job, keep them defined and ready to deploy.
Tools
Timely deployment is dependent on having all appropriate tools needed to respond to a situation. Assemble an emergency kit or “go bag” that includes everything the team might need. Keep it in a well-known, readily accessible location and periodically check to make sure it is complete. Some things that should be included in every go bag are common screwdrivers and network maintenance tools, special hardware analyzers, updated copies of diagnostic software, a copy of your Incident Response Plan, incident handler’s checklists, team member contact list, USB drives, bootable discs and drives for image capture, a laptop with forensic and anti-malware software, and a journal for documentation.
Training
The last component of your Incident Response Plan could very well prove to be the most important. Conduct regular training with your team and periodically invite a few other members of the organization to attend in order help build a companywide culture of security. During these training meetings, you will need to review and discuss lessons learned from previous incident documentation. Finalize your discoveries in a report that includes the scope of the problem, how it was contained and eradicated, recovery steps taken, and areas where the team was effective and where it might improve. Determine if the Incident Recovery Plan needs to be revised considering these facts. It is critical to actually train during these meetings as well. Run table-top simulations and live drills of different incidents that could occur. Make sure you involve every member of the team, and get participation from everyone. This training must be taken seriously in order to ensure the team is ready to act when it matters most, and in order to find weaknesses in the Incident Response Plan that need to be addressed.
If you still believe that your organization doesn’t need an Incident Response Plan, just try running a live drill without one, and you’ll see how badly things go. If you’re too scared to perform such an exercise because of what weaknesses it might reveal, just imagine how much worse it will be when the inevitable happens for real and you’re unprepared.
