|By PagerDuty Blog||
|February 22, 2017 01:30 AM EST||
Fast Data, Fast Monitoring
By Christopher Tozzi
Big data is old news. Today, the key to leveraging data effectively is to do fast data.
In a similar fashion, traditional incident management-which entails collecting and analyzing large volumes of monitoring information-is no longer enough. Organizations must also now do "fast monitoring," which means not only collecting monitoring data; but making it actionable in real-time.
This post examines what fast monitoring means, and explains how incident management teams can implement this approach to realize great benefits.
Defining Fast Data
To understand the concept of fast monitoring, you have to understand fast data - one of the newest innovations in the big data world.
Very simply put, fast data is big data done in a fast way. Whereas big data traditionally meant storing large amounts of information and analyzing it later, fast data means performing data analytics on large amounts of information as quickly as possible-ideally, in real-time. The goal is to analyze the data when it is as actionable and relevant as possible.
Being able to stream data from its source into an analytics platform is an important part of leveraging fast data. This is why big data tools like Apache Spark have become popular in recent years. By supporting streaming data collection, as well as in-memory processing, Spark can ingest and analyze large amounts of information at speeds much faster than non-streaming, on-disk data analytics platforms.
Fast Data and Incident Management
Incident management is a different field from data analytics, but incident management admins can learn a lot from the fast data trend. In the infrastructure monitoring and incident management world, being able to analyze large amounts of monitoring and alert data in real-time to improve response is now more important than ever.
From traditional incident management to fast incident management
The connection between fast data and fast monitoring is not a coincidence. In many ways, the evolution of incident management mirrors the evolution of data analytics.
Until about ten years ago, data, like infrastructure, was relatively small. There was no need for most organizations to analyze petabytes of data because they didn't generate that much. Similarly, most organizations had no need for monitoring solutions that could support large and diverse infrastructures. They could instead get away with basic monitoring systems to keep track of relatively small and uncomplicated networks of servers and workstations.
Then, in the mid-2000s, both data and infrastructure started getting much bigger. The digitization of everything meant that organizations started collecting reams of information, giving rise to big data. Meanwhile, the proliferation of mobile devices, the rise of virtualization and the need for ever-more computing power made infrastructure much larger and more complicated. This new landscape required big monitoring.
And within the past few years, another sea of change has taken place. In an age when information is constantly changing, performing analytics on data that is even just a few hours old undercuts the value of analytics. Similarly, performing incident management on the basis of monitoring information that is not up-to-date prevents admins from triaging and responding to incidents effectively.
So, while fast data and fast monitoring may require different toolsets, the principles and motivations behind both trends are the same. Incident management teams seeking to keep infrastructure and apps running as smoothly as possible would do well to emulate their data-analyst colleagues by focusing on fast monitoring.
Facilitating Fast Monitoring
Collecting and reacting to monitoring information quickly may sound simple enough - but how can you make fast monitoring work in practice? The major guidelines to follow include:
- Centralize data collection. To make sense of monitoring information as quickly as possible, you should forward all monitoring data to a central interface. This will obviate the need to switch between different dashboards or monitoring systems, which wastes time and mental energy as well as makes it highly difficult to understand root cause.
- Collect all information available. Traditional incident management tends to focus just on collecting machine data and alerts. That information provides part of what you need to do fast monitoring, but in order to react as quickly as possible to incidents, you should have as broad a scope as possible of visibility and insight. For instance, gathering human-generated data from tickets and support calls should not be ignored. This also means taking advantage of features like PagerDuty's Custom Event Transformer in order to collect data from sources, such as social media channels, that are not traditionally a part of incident management workflows.
- Minimize noise. You're going to have a large amount of alerts, but only some of them require action. It's therefore absolutely crucial to cut out the noise, and suppress anything that is non-actionable, so that the number of alerts you have to pay attention to is minimal. Alerts should automatically be de-duplicated, and it should be easy to group related symptoms into a single issue that requires resolution. This will facilitate instantaneous identification of the alerts that require attention, and trigger the appropriate response workflow in real-time.
- Make data easy to interpret. Collecting large amounts of monitoring data and storing it in a central location will help you to turn that data into value quickly. However, to streamline the process fully, you also should make sure that data from various sources is normalized into a consistent format in order to promote easy analysis of all of the information on the dashboard and reduce cognitive load. That way, you don't need to memorize or contextualize all the schema from different vendors. To do this, you need incident management solutions that can take information in diverse forms and normalize fields universally, to generate immediately actionable, easy-to-understand insights.
All of these practices minimize the amount of manual analysis required from incident management admins during a crucial incident. In turn, they minimize the time between alert collection and action, making it feasible for incident management staff to react to incidents as quickly as they occur, and truly turn fast monitoring into real-time response for improved uptime.
Delaney, Ozzy. "Speeding." Jan 20, 2015. Online image. <https:[email protected]/15854782234/>
@DevOpsSummit at Cloud Expo taking place June 6-8, 2017, at Javits Center, New York City, and is co-located with the 20th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world.
DevOps at Cloud Expo / @ThingsExpo 2017 New York
(June 6-8, 2017, Javits Center, Manhattan)
DevOps at Cloud Expo / @ThingsExpo 2017 Silicon Valley
(October 31 - November 2, 2017, Santa Clara Convention Center, CA)
Download Show Prospectus ▸ Here
The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
@DevOpsSummit will expand the DevOps community, enable a wide sharing of knowledge, and educate delegates and technology providers alike. Recent research has shown that DevOps dramatically reduces development time, the amount of enterprise IT professionals put out fires, and support time generally. Time spent on infrastructure development is significantly increased, and DevOps practitioners report more software releases and higher quality. Sponsors of @DevOpsSummit will benefit from unmatched branding, profile building and lead generation opportunities through:
- Featured on-site presentation and ongoing on-demand webcast exposure to a captive audience of industry decision-makers.
- Showcase exhibition during our new extended dedicated expo hours
- Breakout Session Priority scheduling for Sponsors that have been guaranteed a 35-minute technical session
- Online advertising in SYS-CON's i-Technology Publications
- Capitalize on our Comprehensive Marketing efforts leading up to the show with print mailings, e-newsletters and extensive online media coverage.
- Unprecedented PR Coverage: Editorial Coverage on DevOps Journal
- Tweetup to over 75,000 plus followers
- Press releases sent on major wire services to over 500 industry analysts.
For more information on sponsorship, exhibit, and keynote opportunities, contact Carmen Gonzalez by email at events (at) sys-con.com, or by phone 201 802-3021.
The World's Largest "Cloud Digital Transformation" Event
@CloudExpo / @ThingsExpo 2017 New York
(June 6-8, 2017, Javits Center, Manhattan)
@CloudExpo / @ThingsExpo 2017 Silicon Valley
(Oct. 31 - Nov. 2, 2017, Santa Clara Convention Center, CA)
Full Conference Registration Gold Pass and Exhibit Hall ▸ Here
Register For @CloudExpo ▸ Here via EventBrite
Register For @ThingsExpo ▸ Here via EventBrite
Register For @DevOpsSummit ▸ Here via EventBrite
Sponsors of Cloud Expo / @ThingsExpo will benefit from unmatched branding, profile building and lead generation opportunities through:
- Featured on-site presentation and ongoing on-demand webcast exposure to a captive audience of industry decision-makers
- Showcase exhibition during our new extended dedicated expo hours
- Breakout Session Priority scheduling for Sponsors that have been guaranteed a 35 minute technical session
- Online targeted advertising in SYS-CON's i-Technology Publications
- Capitalize on our Comprehensive Marketing efforts leading up to the show with print mailings, e-newsletters and extensive online media coverage
- Unprecedented Marketing Coverage: Editorial Coverage on ITweetup to over 100,000 plus followers, press releases sent on major wire services to over 500 industry analysts
For more information on sponsorship, exhibit, and keynote opportunities, contact Carmen Gonzalez (@GonzalezCarmen) today by email at events (at) sys-con.com, or by phone 201 802-3021.
All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Track 1. FinTech
Track 2. Enterprise Cloud | Digital Transformation
Track 3. DevOps, Containers & Microservices
Track 4. Big Data | Analytics
Track 5. Industrial IoT
Track 6. IoT Dev & Deploy | Mobility
Track 7. APIs | Cloud Security
Track 8. AI | ML | DL | Cognitive Computing
Delegates to Cloud Expo / @ThingsExpo will be able to attend 8 simultaneous, information-packed education tracks.
There are over 120 breakout sessions in all, with Keynotes, General Sessions, and Power Panels adding to three days of incredibly rich presentations and content.
Join Cloud Expo / @ThingsExpo conference chair Roger Strukhoff (@IoT2040), June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA for three days of intense Enterprise Cloud and 'Digital Transformation' discussion and focus, including Big Data's indispensable role in IoT, Smart Grids and (IIoT) Industrial Internet of Things, Wearables and Consumer IoT, as well as (new) Digital Transformation in Vertical Markets.
Financial Technology - or FinTech - Is Now Part of the @CloudExpo Program!
Accordingly, attendees at the upcoming 20th Cloud Expo / @ThingsExpo June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA will find fresh new content in a new track called FinTech, which will incorporate machine learning, artificial intelligence, deep learning, and blockchain into one track.
Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expensive intermediate processes from their businesses.
FinTech brings efficiency as well as the ability to deliver new services and a much improved customer experience throughout the global financial services industry. FinTech is a natural fit with cloud computing, as new services are quickly developed, deployed, and scaled on public, private, and hybrid clouds.
More than US$20 billion in venture capital is being invested in FinTech this year. @CloudExpo is pleased to bring you the latest FinTech developments as an integral part of our program, starting at the 20th International Cloud Expo June 6-8, 2017 in New York City and October 31 - November 2, 2017 in Silicon Valley.
The upcoming 20th International @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA announces that its Call For Papers for speaking opportunities is open.
Submit your speaking proposal today! ▸ Here
Our Top 100 Sponsors and the Leading "Digital Transformation" Companies
(ISC)2, 24Notion (Bronze Sponsor), 910Telecom, Accelertite (Gold Sponsor), Addteq, Adobe (Bronze Sponsor), Aeroybyte, Alert Logic, Anexia, AppNeta, Avere Systems, BMC Software (Silver Sponsor), Bsquare Corporation (Silver Sponsor), BZ Media (Media Sponsor), Catchpoint Systems (Silver Sponsor), CDS Global Cloud, Cemware, Chetu Inc., China Unicom, Cloud Raxak, CloudBerry (Media Sponsor), Cloudbric, Coalfire Systems, CollabNet, Inc. (Silver Sponsor), Column Technologies, Commvault (Bronze Sponsor), Connect2.me, ContentMX (Bronze Sponsor), CrowdReviews (Media Sponsor) CyberTrend (Media Sponsor), DataCenterDynamics (Media Sponsor), Delaplex, DICE (Bronze Sponsor), EastBanc Technologies, eCube Systems, Embotics, Enzu Inc., Ericsson (Gold Sponsor), FalconStor, Formation Data Systems, Fusion, Hanu Software, HGST, Inc. (Bronze Sponsor), Hitrons Solutions, IBM BlueBox, IBM Bluemix, IBM Cloud (Platinum Sponsor), IBM Cloud Data Services/Cloudant (Platinum Sponsor), IBM DevOps (Platinum Sponsor), iDevices, Industrial Internet of Things Consortium (Association Sponsor), Impinger Technologies, Interface Masters, Intel (Keynote Sponsor), Interoute (Bronze Sponsor), IQP Corporation, Isomorphic Software, Japan IoT Consortium, Kintone Corporation (Bronze Sponsor), LeaseWeb USA, LinearHub, MangoApps, MathFreeOn, Men & Mice, MobiDev, New Relic, Inc. (Bronze Sponsor), New York Times, Niagara Networks, Numerex, NVIDIA Corporation (AI Session Sponsor), Object Management Group (Association Sponsor), On The Avenue Marketing, Oracle MySQL, Peak10, Inc., Penta Security, Plasma Corporation, Pulzze Systems, Pythian (Bronze Sponsor), Cosmos, RackN, ReadyTalk (Silver Sponsor), Roma Software, Roundee.io, Secure Channels Inc., SD Times (Media Sponsor), SoftLayer (Platinum Sponsor), SoftNet Solutions, Solinea Inc., SpeedyCloud, SSLGURU LLC, StarNet, Stratoscale, Streamliner, SuperAdmins, TechTarget (Media Sponsor), TelecomReseller (Media Sponsor), Tintri (Welcome Reception Sponsor), TMCnet (Media Sponsor), Transparent Cloud Computing Consortium, Veeam, Venafi, Violin Memory, VAI Software, Zerto
About SYS-CON Media & Events
SYS-CON Media (www.sys-con.com) has since 1994 been connecting technology companies and customers through a comprehensive content stream - featuring over forty focused subject areas, from Cloud Computing to Web Security - interwoven with market-leading full-scale conferences produced by SYS-CON Events. The company's internationally recognized brands include among others Cloud Expo® (@CloudExpo), Big Data Expo® (@BigDataExpo), DevOps Summit (@DevOpsSummit), @ThingsExpo® (@ThingsExpo), Containers Expo (@ContainersExpo) and Microservices Expo (@MicroservicesE).
Cloud Expo®, Big Data Expo® and @ThingsExpo® are registered trademarks of Cloud Expo, Inc., a SYS-CON Events company.
Feb. 22, 2017 09:45 AM EST Reads: 1,306
Feb. 22, 2017 09:45 AM EST Reads: 781
Feb. 22, 2017 09:29 AM EST
Feb. 22, 2017 09:15 AM EST Reads: 1,369
Feb. 22, 2017 08:45 AM EST Reads: 1,122
Feb. 22, 2017 08:30 AM EST Reads: 837
Feb. 22, 2017 08:30 AM EST Reads: 820
Feb. 22, 2017 08:15 AM EST Reads: 4,649
Feb. 22, 2017 08:00 AM EST Reads: 656
Feb. 22, 2017 07:30 AM EST Reads: 296
Feb. 22, 2017 07:00 AM EST Reads: 2,912
Feb. 22, 2017 06:45 AM EST Reads: 1,616
Microservices are a very exciting architectural approach that many organizations are looking to as a way to accelerate innovation. Microservices promise to allow teams to move away from monolithic "ball of mud" systems, but the reality is that, in the vast majority of organizations, different projects and technologies will continue to be developed at different speeds. How to handle the dependencies between these disparate systems with different iteration cycles? Consider the "canoncial problem" ...
Feb. 22, 2017 06:00 AM EST Reads: 5,253
Both SaaS vendors and SaaS buyers are going “all-in” to hyperscale IaaS platforms such as AWS, which is disrupting the SaaS value proposition. Why should the enterprise SaaS consumer pay for the SaaS service if their data is resident in adjacent AWS S3 buckets? If both SaaS sellers and buyers are using the same cloud tools, automation and pay-per-transaction model offered by IaaS platforms, then why not host the “shrink-wrapped” software in the customers’ cloud? Further, serverless computing, cl...
Feb. 22, 2017 06:00 AM EST Reads: 1,827
“We're a global managed hosting provider. Our core customer set is a U.S.-based customer that is looking to go global,” explained Adam Rogers, Managing Director at ANEXIA, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Feb. 22, 2017 05:45 AM EST Reads: 1,689