![\"\"](\"https:\/\/estatus.io\/blog\/wp-content\/uploads\/2017\/10\/timeisnow-min-300x251.jpg\")
<\/p>\n<\/p>\n
Are you ready for the next outage or degradation in performance? \u00a0Do you have your incident response plan<\/strong> in place? During the last few years, we have witnessed several instances when massive service providers have experienced outages that have cost companies billions.<\/p>\n Let’s take for instance Amazon\u2019s S3 outage<\/a> earlier this year (February 28, 2017). Some estimate that outage alone, lasting only a few hours, could have cost upwards of $150 million. One single service disruption caused degradations and disruptions to 48 other internal services they have listed on their service health dashboard. One of the lessons to be learned from this event. They host their own status page, which itself was impacted by the outage, resulting in the status indicators not displaying the red outage icon. As a result of not having their page hosted by a third party, it put additional stress on their support team.<\/p>\n This is a continuous risk of modern IT organizations, as we have become dependent on SaaS services. \u00a0The days of hosting your own complex datacenters are over and the future of IT looks to integrate existing services in a meaningful way.<\/p>\n Modern services and applications are typically built on top of a backbone that is hosted offsite and out of the IT organization’s control. \u00a0These great technological dependencies have created a need for these services to be available 24\/7\/365 and we have seen what can happen when they are not. \u00a0I am willing to bet that you have several services that either partially or solely depends on a third party vendor.<\/p>\n Modern business means that you are providing some sort of IT service to your customers or end users. \u00a0This is consistent no matter the size of your business and the number of your employees you have. \u00a0From the diner on the corner to the tech giant, all businesses are dependent upon IT services.<\/p>\n Let\u2019s think about this for a minute and think about the small diner on the corner. \u00a0There are a lot of logistics behind the scenes for this diner to operate. \u00a0They must have Internet connectivity to order food and supplies. \u00a0They must have an Internet-connected POS system to accept credit card payments (who has cash anymore). \u00a0This POS system is also tracking what was ordered and therefore integrated with an inventory control system.<\/p>\n This diner must be ready for an incident just as much as your tech giant SaaS provider that has millions of clients. \u00a0What will they do when these services fail? \u00a0How will they notify their staff? \u00a0How will they notify their customers? \u00a0What secondary systems are in place in case the first ones fail? \u00a0\u00a0These things can be planned before there are problems and detailed in your incident response plan. \u00a0The time to take action is not when you are in crisis mode, you should be prepared for the day when your systems will fail.<\/p>\n The Incident Response planning phase is the most intensive and important aspects of your response plan. \u00a0This is the stage that you are going to lay the foundation and put in all the hard work to develop your plan, process and procedures. This is the kickoff point and is critical to your overall success.<\/p>\n During the planning phase, you should develop training and make sure that all of your staff attends the same training. \u00a0It is important that all of your employees have the same incident response training and are familiar with the various roles that they may fill. \u00a0This is also a good time to plan any drills to test your processes throughout the lifecycle of your plan.<\/p>\n As a matter of practice, you should host bi-annual drills within your organization to prepare your staff for an outage. \u00a0This will ensure that your staff gets an opportunity to experience an outage in the sandbox before the stressful real-world incident. \u00a0Furthermore, spacing these 6 months apart will account for any attrition that you have within your organization and will be a continuation of the onboarding training that your staff should be receiving.<\/p>\n The popular ITIL framework defines an Incident as an unplanned interruption to an IT Service or reduction in the quality of an IT service. \u00a0This is a broad definition and there are typically other factors that go into your incident response decisions:<\/p>\n Here are some factors that should be considered:<\/p>\n There are many components to your incident response plan and these should include:<\/p>\n There a many moving parts within your incident response procedures but the role of your staff are most critical to your success. \u00a0The following roles should be considered and applied to your plan:<\/p>\n The Incident Commander is responsible for the incident and has several duties throughout the lifecycle of an incident. \u00a0He or she should know what is expected of them during this role and proper proactive training should be given throughout your company. \u00a0Training should not happen during a live incident.<\/p>\n The Incident Commander is in charge of all of the moving pieces and is making sure that everyone is staying on task. \u00a0These duties include but are not limited to:<\/p>\n The incident response team is responsible for the identification, troubleshooting and ultimate resolution of your incident. \u00a0These teams can be small or large and will vary between organizations and industry. \u00a0However, the key to having a great incident response team is to make certain that you have enough resources to handle any incident that may arise.<\/p>\n There are some key positions that should be included on your incident response team:<\/p>\n Your service desk is your hub and the tip of the iceberg for your organization. \u00a0Typically all of the work that flows through your organization is originated at the service desk. \u00a0This is also the place that users will contact when you are experiencing an outage. \u00a0It is important that your service desk is aware of your outage and any workarounds that have been identified throughout the identification or troubleshooting processes.<\/p>\n I cannot reiterate this enough that you must keep your service desk informed during any outage. \u00a0I have seen the consequences notifying the service desk as an afterthought and it is a recipe for a disorganized chaotic mess. Build this into your plan and make sure you have mechanisms in place to make these critical notifications.<\/p>\n Communication during an outage is critical to your overall success. \u00a0Too little or too late and you have a herd of angry customers banging down your doors. \u00a0\u00a0Too much communication – wait that never happens so we can skip that. \u00a0A designated communications representative is imperative to make sure that you are making timely notifications at an acceptable cadence to your customers based upon your incident response plan.<\/p>\n While often a separate role, this role can be shared on within your response team. However, this role should be hyper-focused on internal and external communication and not have to share the burden of troubleshooting, etc. \u00a0To avoid any missteps it is recommended that this resource is free to manage the communications process.<\/p>\n Your various team and department management should be available during a problem but should not interfere in the incident response steps the team needs to carry out. There are a lot of critical decisions that may arise during an incident and management should be on hand to aid in these decisions or approve any emergency changes, etc.<\/p>\n During an incident, it is critical to alert your internal support and technical teams. \u00a0These are the people that need to start diagnosing the Incident and moving towards a resolution. \u00a0This is typically the most critical piece of your plan and should involve a solution to notify all of your technical teams and use a swarming technique to address the incident.<\/p>\n This swarming technique breaks through the typical IT silos and brings all of your technical experts together to quickly identify root cause and move towards resolution. \u00a0Your technical teams will work collaboratively and not have to worry about a strict chain of command during the identification of root cause. \u00a0This leads to faster root cause identification and ultimate incident resolution.<\/p>\n The next critical piece of communication is to your customers or end-users and this can take place simultaneously to your internal communication or immediately after. \u00a0This will vary from business to business and other incident impact and urgency factors will go into this determination. \u00a0Find a solution that works for you and your business.<\/p>\n Customers will start flooding your service desk or business phone at the first sign of a problem and this can cost a company millions during major outages. \u00a0In 2017 we should be providing proactive communications to users and avoiding the costly IT phone call to report an issue that we already know is happening. There are several communication methods that may work for your business:<\/p>\n The goal here is to get the information in front of your customers so that they are informed and so that they are not flooding your IT help desk with costly support calls. \u00a0Furthermore, studies show that proactive notification methods reduce customer stress and instill confidence.<\/p>\n A company status page is how modern organizations communicate with their customers and this will be discussed in greater detail later in this article. \u00a0The comprehensive solutions provide a status page and communication platform that your customers will love. \u00a0Consider your plan before you next outage strikes.<\/p>\n Time metrics are critical to your incident response policy and give you a benchmark for making decisions. \u00a0You should assign time limits to various events and have an action plan to execute once these thresholds have been met or exceeded. \u00a0For example, maybe you have an internal incident triage time of 15 minutes. \u00a0During this initial 15 minutes, your support teams are identifying the problem and trying to get a grasp on the impact. \u00a0At the 15 minutes mark, you should be sending your external communication or activating the necessary communication channels.<\/p>\n![\"Amazon's](\"https:\/\/estatus.io\/blog\/wp-content\/uploads\/2017\/10\/amazon-outage.png\")
<\/p>\nWho Needs an Incident Response Plan?<\/h1>\n
Incident Response Planning<\/h2>\n
Incident Response Process & Considerations:<\/h2>\n
What is an incident?<\/h3>\n
\n
Incident Response<\/h3>\n
\n
Incident Response Roles<\/h2>\n
![\"incident](\"https:\/\/estatus.io\/blog\/wp-content\/uploads\/2017\/10\/incidente-response-roles-min.png\")
<\/a><\/p>\n\n
Incident Commander<\/h3>\n
\n
Incident Response Team<\/h3>\n
![\"Who](\"https:\/\/estatus.io\/blog\/wp-content\/uploads\/2017\/10\/incident-response-team-min.jpg\")
<\/p>\n\n
IT Service Desk<\/h3>\n
Communications<\/h3>\n
![\"Is](\"https:\/\/estatus.io\/blog\/wp-content\/uploads\/2017\/10\/redphone-min.png\")
<\/p>\nManagement<\/h3>\n
Internal Communication Plan<\/h2>\n
![\"We](\"https:\/\/estatus.io\/blog\/wp-content\/uploads\/2017\/10\/phone2-min.png\")
<\/p>\nExternal Communication Plan<\/h2>\n
\n
Time Metrics<\/h2>\n
Status Page<\/h2>\n