Blog Feed Post

How to Prevent Alerting Overload

In our always-on, IoT-enabled, cloud-connected, big data age, we face a major paradox: it’s now easier than ever to collect large amounts of data — yet the more data we collect, the harder it becomes to monitor situations effectively.

This problem is similar to what psychologists call “information overload” — the phenomenon that causes someone to fail to make decisions effectively because he has too much information to contend with.

In some contexts information overload is unavoidable. If you get hundreds of emails each day, there may not be much you can do about feeling overwhelmed by them, as you don’t necessarily have a lot of control over who sends you an email. Yet, when it comes to data center infrastructure, information overload is not inevitable. It’s entirely up to you to decide how much and what types of data to collect. If you find that you have too much data to parse feasibly, it means you need to rethink your monitoring practices and alert filtering.

Of course, as we’ve already noted, many admins may find themselves fighting an uphill battle when it comes to preventing information overload in the data center. That’s because the explosion of the cloud and the advent of IoT — and all of the inexpensive data that comes alongside those trends — have made it easier than ever to collect all manners of information about your servers and applications.

What’s Critical, What’s Not

That’s why it’s now more important than ever to decide which types of monitoring you actually need, what to set up notifications on, and what you can do without. Just because adding more monitoring to your infrastructure is easy and inexpensive doesn’t mean you should necessarily do it.

If you add monitoring blindly, you’re shooting yourself in the foot by collecting more data than you can ever process or act on effectively. This turns into fatigue for your on-call staff, wasted time spent on low priority issues, and causes low priority issues to distract from the critical ones.

Successful alert management depends on your particular needs, of course. There’s no one-size-fits-all approach. In general, it’s a good idea to try to restrict yourself to deploying sensors that center around the following types of information:

  • Security incidents: You’ll want to be alerted to things like repeated failed login attempts or port scans so you can stay ahead of threats.
  • Host failure: If a physical or virtual server fails to start, or crashes suddenly, that’s an important event to know about.
  • Resource exhaustion: You don’t want to wait until you run out of data storage or network bandwidth to find out that you should be adding more. Use sensors to warn you when usage starts to approach the maximum available and stays at that level for more than a short time.

Again, your mileage may well vary. But the above list provides the core essential types of events you should be notified on.

Monitoring vs. Alarms

There are other types of data that are good to monitor but may not require an alarm. Those include things like:

  • CPU usage: This can vary widely throughout the day due to a number of factors. You want to know about general trends, but you don’t need an alarm to tell you each time CPU usage has jumped.
  • Network load: This is in the same category as CPU usage. Network load varies naturally. You should know your data center’s trends so you can plan for long-term expansion. But there’s no need to set off alarms just because a lot of devices happen to be on the network in a given moment — unless, of course, the situation is extreme and sustained.
  • Environmental conditions: You should track things like data center temperature. But this is the type of incident that can usually be handled in an automated fashion. Instead of having sensors send you an alert when temps climb high, have software that turns up the cooling units for you. You only need an alert if temperatures approach critical level and stay there.

It’s quite possible that an issue triggered by a sensor like processor queue length can easily be covered indirectly with the more relevant data point such as processor utilization.

The Right Data for the Right People

Another way to make sure you’re getting optimal results from your sensors is to make sure the right incident notifications are going to the right people.

Platforms like PagerDuty let you specify an order of command for handling different types of events. Rather than blanketing your whole team with incident notifications, make sure only the exact right people who need to handle issues get woken up. This minimizes unplanned work and alert fatigue in responding to issues.

You can also configure PagerDuty to send notifications to a larger group if the initial recipients don’t respond in a certain amount of time.

Get More Out of Logs

Last but not least, keep in mind that there are lots of different ways to deal with information. One way is to generate alerts. But another is to use log analytics tools to identify trends that stretch across a large amount of data collected by various monitoring tools.

By boiling your log results down to the essentials, you can figure out what you should be paying attention to without having to handle a huge number of events on an individual basis.

That’s why PagerDuty offers features like integrations with Splunk and other analysis tools. These are ideal for providing a way to derive value from monitoring data without suffering information overload. 

The post How to Prevent Alerting Overload appeared first on PagerDuty.

Read the original blog entry...

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

Latest Stories
"ZeroStack is a startup in Silicon Valley. We're solving a very interesting problem around bringing public cloud convenience with private cloud control for enterprises and mid-size companies," explained Kamesh Pemmaraju, VP of Product Management at ZeroStack, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"Codigm is based on the cloud and we are here to explore marketing opportunities in America. Our mission is to make an ecosystem of the SW environment that anyone can understand, learn, teach, and develop the SW on the cloud," explained Sung Tae Ryu, CEO of Codigm, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
High-velocity engineering teams are applying not only continuous delivery processes, but also lessons in experimentation from established leaders like Amazon, Netflix, and Facebook. These companies have made experimentation a foundation for their release processes, allowing them to try out major feature releases and redesigns within smaller groups before making them broadly available. In his session at 21st Cloud Expo, Brian Lucas, Senior Staff Engineer at Optimizely, discussed how by using ne...
"CA has been doing a lot of things in the area of DevOps. Now we have a complete set of tool sets in order to enable customers to go all the way from planning to development to testing down to release into the operations," explained Aruna Ravichandran, Vice President of Global Marketing and Strategy at CA Technologies, in this SYS-CON.tv interview at DevOps Summit at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
Enterprises are moving to the cloud faster than most of us in security expected. CIOs are going from 0 to 100 in cloud adoption and leaving security teams in the dust. Once cloud is part of an enterprise stack, it’s unclear who has responsibility for the protection of applications, services, and data. When cloud breaches occur, whether active compromise or a publicly accessible database, the blame must fall on both service providers and users. In his session at 21st Cloud Expo, Ben Johnson, C...
Data scientists must access high-performance computing resources across a wide-area network. To achieve cloud-based HPC visualization, researchers must transfer datasets and visualization results efficiently. HPC clusters now compute GPU-accelerated visualization in the cloud cluster. To efficiently display results remotely, a high-performance, low-latency protocol transfers the display from the cluster to a remote desktop. Further, tools to easily mount remote datasets and efficiently transfer...
"Infoblox does DNS, DHCP and IP address management for not only enterprise networks but cloud networks as well. Customers are looking for a single platform that can extend not only in their private enterprise environment but private cloud, public cloud, tracking all the IP space and everything that is going on in that environment," explained Steve Salo, Principal Systems Engineer at Infoblox, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventio...
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Agile has finally jumped the technology shark, expanding outside the software world. Enterprises are now increasingly adopting Agile practices across their organizations in order to successfully navigate the disruptive waters that threaten to drown them. In our quest for establishing change as a core competency in our organizations, this business-centric notion of Agile is an essential component of Agile Digital Transformation. In the years since the publication of the Agile Manifesto, the conn...