Welcome!

Blog Feed Post

5 Incident Management Tools You Need During a Firefight

It’s critical to have the right tools in place before a firefight happens. A lack of proper tooling makes it significantly more difficult to recognize, organize, fight, and resolve a major outage. This is especially true when teams are busy fighting rather than communicating to internal and external stakeholders. If best practices have been established ahead of time, a difficult incident can be handled much more smoothly.

The following is not an exhaustive list of domains to plan prior to an outage, but they will greatly improve your organization’s ability to coordinate and be prepared for any issue.

1. Internal Communications

Internal communication will commonly take place in email. This is problematic for a number of reasons. Email is a one-to-one medium. It defaults closed, meaning it is only readable for the sender and receiver(s), and is inherently bulky and difficult to parse through when quick status information is needed. Persistent collaboration environments like Slack and HipChat provide an externally hosted location to disseminate information. Both of these platforms also provide public, optional subscribe, topical channels that can be used to disseminate information. At the critical level, status updates (or messaging that the issue is already known and being worked on) can be provided to key staff (support, leadership) in near real-time.

2. Application Performance and Infrastructure Monitoring

Ideally, the team will know there is an issue with an application before the customer does. Application and infrastructure monitoring technology can help ensure this is the case and can provide valuable information in the midst of the outage as to whether a fix or update is working as it should (New Relic for application monitoring and AWS CloudWatch are two such technologies). It is also important to monitor both application performance and infrastructure performance, and (ideally) link the two together, with a solution such as PagerDuty, to consolidate all service health data into a single view and notify the on-call resource if any issue requires urgent action. It is much easier to troubleshoot an issue if you have visibility to both layers and can identify the root cause.

3. Status Updates

When there is a performance issue, support teams will be inundated with requests for updates. Key ways to mitigate this influx are via Twitter, a status page, or to engage business stakeholders with a product like PagerDuty. These are separate from your primary infrastructure and should be resilient to even site-wide outages. On Twitter, users can easily look for pinned tweets and recent replies if they are having an issue. Users can also check statusapp.com for any “yellow” or “red” statuses. An easy-to-read status page like the one from statuspage.io is a critical component to disseminate information to your customers during an outage. A user will build trust in the page if it is accurate and includes updates for minor disruptions — and in that way, they also build more trust in your business. It should also contain updates when an issue is undergoing troubleshooting, and include status for each major subcomponent. These updates should be available within minutes, for complete visibility. Finally, with capabilities like PagerDuty’s Stakeholder Engagement, any incident responder can easily send out a status update that reaches predefined groups of business stakeholders via any preferred notification channel — phone, SMS, email, or push notification. Stakeholders can also subscribe to incident status updates to get real-time information on any issue that is customer-impacting.

4. Ticketing Solution

A ticketing solution like ZenDesk is absolutely critical to managing an outage. A significant outage can be highly disruptive and forfeit substantial goodwill. A ticket management system will help to identify intermittent issues an application monitor may have missed. It will also track and disseminate information relative to an influx of support requests. Workflows for issue escalation will raise potential issues more quickly than relying on individual judgement, especially on larger support teams. Ready-made message templates will help keep messaging consistent and accurate during an outage, and “related to” tags will also make it easier to debrief an issue once it has been resolved.

5. Procedure Tracking

With proper procedures in place, an organization can anticipate issues that are likely to arise from their applications. These scenarios should be documented ahead of time. Troubleshooting, mitigation, and remediation information should be documented and surfaced for the team. The procedure can also include a checklist of duties — one that lays out who does what, and includes emergency numbers and who is on-call. If resources are available, a tabletop exercise of a mock outage is extremely helpful in identifying gaps before a major outage occurs. Then after a firefight has occurred, debrief with the team in a post-mortem and improve your procedures. There will be another outage, and any additional information you can add to your process will speed recovery. As with the other above items, it is possible your local architecture will become unavailable, so storing these procedures on an externally hosted repository, or automating it with a solution such as PagerDuty, is preferred.

These tools are only an initial list. Their effectiveness in an outage is only as valuable as the time that was spent to properly configure and understand them ahead of time. Communicating with both internal and external stakeholders is key in any firefight, as much within IT as in any other function or industry.

The post 5 Incident Management Tools You Need During a Firefight appeared first on PagerDuty.

Read the original blog entry...

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

Latest Stories
Containers are rapidly finding their way into enterprise data centers, but change is difficult. How do enterprises transform their architecture with technologies like containers without losing the reliable components of their current solutions? In his session at @DevOpsSummit at 21st Cloud Expo, Tony Campbell, Director, Educational Services at CoreOS, will explore the challenges organizations are facing today as they move to containers and go over how Kubernetes applications can deploy with lega...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, will provide a fun and simple way to introduce Machine Leaning to anyone and everyone. Together we will solve a machine learning problem and find an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intellige...
Today most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes significant work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reducti...
We all know that end users experience the Internet primarily with mobile devices. From an app development perspective, we know that successfully responding to the needs of mobile customers depends on rapid DevOps – failing fast, in short, until the right solution evolves in your customers' relationship to your business. Whether you’re decomposing an SOA monolith, or developing a new application cloud natively, it’s not a question of using microservices – not doing so will be a path to eventual b...
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It’s clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. Tha...
As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...
SYS-CON Events announced today that Ryobi Systems will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ryobi Systems Co., Ltd., as an information service company, specialized in business support for local governments and medical industry. We are challenging to achive the precision farming with AI. For more information, visit http:...
Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...
As you move to the cloud, your network should be efficient, secure, and easy to manage. An enterprise adopting a hybrid or public cloud needs systems and tools that provide: Agility: ability to deliver applications and services faster, even in complex hybrid environments Easier manageability: enable reliable connectivity with complete oversight as the data center network evolves Greater efficiency: eliminate wasted effort while reducing errors and optimize asset utilization Security: imple...
High-velocity engineering teams are applying not only continuous delivery processes, but also lessons in experimentation from established leaders like Amazon, Netflix, and Facebook. These companies have made experimentation a foundation for their release processes, allowing them to try out major feature releases and redesigns within smaller groups before making them broadly available. In his session at 21st Cloud Expo, Brian Lucas, Senior Staff Engineer at Optimizely, will discuss how by using...
The next XaaS is CICDaaS. Why? Because CICD saves developers a huge amount of time. CD is an especially great option for projects that require multiple and frequent contributions to be integrated. But… securing CICD best practices is an emerging, essential, yet little understood practice for DevOps teams and their Cloud Service Providers. The only way to get CICD to work in a highly secure environment takes collaboration, patience and persistence. Building CICD in the cloud requires rigorous ar...
In this strange new world where more and more power is drawn from business technology, companies are effectively straddling two paths on the road to innovation and transformation into digital enterprises. The first path is the heritage trail – with “legacy” technology forming the background. Here, extant technologies are transformed by core IT teams to provide more API-driven approaches. Legacy systems can restrict companies that are transitioning into digital enterprises. To truly become a lead...
Companies are harnessing data in ways we once associated with science fiction. Analysts have access to a plethora of visualization and reporting tools, but considering the vast amount of data businesses collect and limitations of CPUs, end users are forced to design their structures and systems with limitations. Until now. As the cloud toolkit to analyze data has evolved, GPUs have stepped in to massively parallel SQL, visualization and machine learning.
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...