Blog Feed Post

Suppress Your Data!

Avoiding Noise in Incident Management

Suppression. According to the thesaurus, this word is synonymous with terms like deletion, elimination, and annihilation.

Yet within the context of incident management, suppression means something quite different. It’s not about getting rid of data forever. It serves instead as a way of making sure that admins focus on the right alerts at the right time by mitigating noise.

Here’s a look at how suppression significantly helps streamline incident management.

Why Suppression is Important

Why is suppression useful in incident management? Simply put, it’s because modern infrastructure generates a huge volume of alerts and admins can’t reasonably expect to be able to review each and every alert. If they try, they will soon become subject to alert fatigue, which means they will begin ignoring potentially important alerts because they are overwhelmed and burned out. And if they stop paying attention to alerts, then the entire incident management process breaks down.https://www.pagerduty.com/wp-content/uploads/2016/11/suppression-300x175... 300w, https://www.pagerduty.com/wp-content/uploads/2016/11/suppression-250x145... 250w, https://www.pagerduty.com/wp-content/uploads/2016/11/suppression-180x105... 180w" sizes="(max-width: 500px) 100vw, 500px" />

Alert suppression is a way of avoiding this issue. By suppressing alerts of certain types, admins can ensure that actionable, high-priority alerts receive the greatest attention. They can also reduce the overall number of alerts that appear on their dashboards, which helps to prevent the risk of alert fatigue.

As an example, consider an organization whose workstations reboot once a week overnight after updates are installed. The reboot would generate a series of alerts as workstations go offline and come back up. Adding these to the incidents dashboard that admins see wouldn’t be helpful, because the alerts in this case reflect a routine procedural event that does not require action. In order to avoid adding this unhelpful noise to admins’ dashboards, admins can configure their incident management software to suppress alerts related to a workstation rebooting.

Suppression: Not an Either/Or Proposition

An important point to understand about alert suppression is that suppressing alerts is not an either/or proposition. In other words, admins’ options are not limited simply to enabling all alerts of a certain type or permanently suppressing all of them.

They can instead take a more nuanced approach to suppression. Alert suppression could be configured in such a way that alerts of a given type are suppressed unless they occur repeatedly within a certain period of time, for example. Alerts could also be configured so that they are reported if they occur during a certain time of day, but are suppressed during other times. Similarly, admins might want to suppress alerts of a particular type if they occur on a certain kind of device, but not others.

This flexibility is important because it ensures that admins can maximize the effectiveness of alerts. Instead of applying broad, blunt suppression policies, they can tweak suppression settings in order to maximize the visibility of important events without adding unnecessary noise to the incident management system.

Nuanced suppression could be helpful in the example above. As I noted, admins generally don’t want to receive alerts when a workstation reboots in the middle of the night following a software update. But if the incident management software detects a workstation that reboots multiple times during the same period, that could signal a problem (like a flawed software update) that admins will want to know about. In this situation, having suppression configured so that only recurring reboots generate incidents that appear in the central dashboard, would help to optimize incident management effectiveness.

Suppression Doesn’t Mean Losing Data

It’s also worth emphasizing that suppression in the context of incident management does not mean that suppressed alerts disappear forever. On the contrary, suppressed alerts still happen, and data related to them should be saved. The only difference between a suppressed alert and a non-suppressed one is that the former is not sent to priority dashboards in the incident management system.

This is important to understand because it means that admins retain the ability to look up suppressed alerts to gain insight into an incident if they need to. This also helps them better tune their alerting thresholds. In addition, suppressed alerts still figure into historical incident management data, which can be used to reveal lots of valuable information about infrastructure efficiency and health trends.

With suppression, then, you get to have your alerts and eat them, too—or something like that.

Suppressed alerts can be leveraged in any way admins need to help identify and respond to incidents, but they don’t clutter dashboards with non-actionable information that gets in the way of resolving incidents that are likely to be of a higher priority. Moreover, suppression can be tweaked so that alerts are suppressed only under exactly the right circumstances, but are always reported so you gain full visibility into your infrastructure.


The post Suppress Your Data! appeared first on PagerDuty.

Read the original blog entry...

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

Latest Stories
Information technology (IT) advances are transforming the way we innovate in business, thereby disrupting the old guard and their predictable status-quo. It’s creating global market turbulence. Industries are converging, and new opportunities and threats are emerging, like never before. So, how are savvy chief information officers (CIOs) leading this transition? Back in 2015, the IBM Institute for Business Value conducted a market study that included the findings from over 1,800 CIO interviews ...
Virtualization over the past years has become a key strategy for IT to acquire multi-tenancy, increase utilization, develop elasticity and improve security. And virtual machines (VMs) are quickly becoming a main vehicle for developing and deploying applications. The introduction of containers seems to be bringing another and perhaps overlapped solution for achieving the same above-mentioned benefits. Are a container and a virtual machine fundamentally the same or different? And how? Is one techn...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
What sort of WebRTC based applications can we expect to see over the next year and beyond? One way to predict development trends is to see what sorts of applications startups are building. In his session at @ThingsExpo, Arin Sime, founder of WebRTC.ventures, will discuss the current and likely future trends in WebRTC application development based on real requests for custom applications from real customers, as well as other public sources of information,
As businesses adopt functionalities in cloud computing, it’s imperative that IT operations consistently ensure cloud systems work correctly – all of the time, and to their best capabilities. In his session at @BigDataExpo, Bernd Harzog, CEO and founder of OpsDataStore, will present an industry answer to the common question, “Are you running IT operations as efficiently and as cost effectively as you need to?” He will expound on the industry issues he frequently came up against as an analyst, and...
Keeping pace with advancements in software delivery processes and tooling is taxing even for the most proficient organizations. Point tools, platforms, open source and the increasing adoption of private and public cloud services requires strong engineering rigor - all in the face of developer demands to use the tools of choice. As Agile has settled in as a mainstream practice, now DevOps has emerged as the next wave to improve software delivery speed and output. To make DevOps work, organization...
ChatOps is an emerging topic that has led to the wide availability of integrations between group chat and various other tools/platforms. Currently, HipChat is an extremely powerful collaboration platform due to the various ChatOps integrations that are available. However, DevOps automation can involve orchestration and complex workflows. In his session at @DevOpsSummit at 20th Cloud Expo, Himanshu Chhetri, CTO at Addteq, will cover practical examples and use cases such as self-provisioning infra...
The financial services market is one of the most data-driven industries in the world, yet it’s bogged down by legacy CPU technologies that simply can’t keep up with the task of querying and visualizing billions of records. In his session at 20th Cloud Expo, Jared Parker, Director of Financial Services at Kinetica, will discuss how the advent of advanced in-database analytics on the GPU makes it possible to run sophisticated data science workloads on the same database that is housing the rich inf...
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
Apache Hadoop is emerging as a distributed platform for handling large and fast incoming streams of data. Predictive maintenance, supply chain optimization, and Internet-of-Things analysis are examples where Hadoop provides the scalable storage, processing, and analytics platform to gain meaningful insights from granular data that is typically only valuable from a large-scale, aggregate view. One architecture useful for capturing and analyzing streaming data is the Lambda Architecture, represent...
My team embarked on building a data lake for our sales and marketing data to better understand customer journeys. This required building a hybrid data pipeline to connect our cloud CRM with the new Hadoop Data Lake. One challenge is that IT was not in a position to provide support until we proved value and marketing did not have the experience, so we embarked on the journey ourselves within the product marketing team for our line of business within Progress. In his session at @BigDataExpo, Sum...
Things are changing so quickly in IoT that it would take a wizard to predict which ecosystem will gain the most traction. In order for IoT to reach its potential, smart devices must be able to work together. Today, there are a slew of interoperability standards being promoted by big names to make this happen: HomeKit, Brillo and Alljoyn. In his session at @ThingsExpo, Adam Justice, vice president and general manager of Grid Connect, will review what happens when smart devices don’t work togethe...
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), will provide an overview of various initiatives to certifiy the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldw...
Adding public cloud resources to an existing application can be a daunting process. The tools that you currently use to manage the software and hardware outside the cloud aren’t always the best tools to efficiently grow into the cloud. All of the major configuration management tools have cloud orchestration plugins that can be leveraged, but there are also cloud-native tools that can dramatically improve the efficiency of managing your application lifecycle.