Related Topics: OpenStack Journal, @DevOpsSummit

OpenStack Journal: Article

Log-level Usage Reports | @DevOpsSummit #DevOps #Docker #Monitoring

The reports can give you an intelligent breakdown of your log volumes by application, server, process

What You Can Learn From Log-level Usage Reports

Separating the signal from the noise is one of the biggest challenges when dealing with machine-generated log data today and has generally required deep technical expertise. However once you find that signal it can be massively useful and can help you make business decisions with a big impact. Today Logentries is announcing log level usage reports, which is one more way we are striving to do the hard work so you don't have to!


At first glance, log-level usage reports might seem quite rudimentary insofar as they are weekly reports of the log volumes you send to Logentries. However, the reports can give you an intelligent breakdown of your log volumes by application, server, process as well as at the individual log level.The usage reports are designed to provide IT and Dev Ops teams with a weekly snapshot of their system activity broken down at a very fine-grained log level giving the ability to see interesting trends in system behavior. In fact, so many of our customers have been asking our newLogentries Data Insights team to run these reports that we've decided to make them widely available. The reports are part of a new service being produced by this team comprised of PhDs and applied research scientists in distributed systems,  and they are supported by Logentries proprietary machine learning and community analytics technology.

Over the past few months we have  used these reports to help a large number of our customers to investigate spikes in system load, potential issues as well as to understand business trends and how they relate to their systems - and might I add with phenomenal results -  based on some really simple analysis. I suppose it doesn't need to be complex to be valuable...

But don't take my word for it...according to Rich Archbold, director of Ops at Intercom

"The new Logentries Log Analytics Reporting service has provided us with insight into our environment that we would never have been able to get before. We now have deep, log-level visibility into application activity, trends, and potential issues on a weekly basis delivered right into our inbox!"

Specifically the usage reports will provide the following:

  • Total number of logs and log sets
  • Total log volume across all logs
  • Top 20 logs by volume
  • Daily breakdown of log volume per log
  • List of inactive logs

Screen Shot 2015-05-07 at 9.55.22 AM

Sample Usage Report
The new reports can be especially useful for:

  • Logging cost analysis: When your log volumes increase significantly you usually want to know why. Looking at your total log volumes will generally not give enough insight to allow you to figure out what has caused the increase or from what part of your system the increase is coming from. However by getting a breakdown of your top 20 logs by volume you can immediately see the log or logs responsible, allowing you to take the next step in determining as to whether this was a result of increased system load or whether this was related to a system issue or rogue process.
  • System load analysis: Log volumes are a great indicator for understanding system load. As load increases so too do your log volumes. Understanding what log(s) are resulting in the increased load will give you better visibility into what part of your system is actually seeing the increased workloads.
  • Troubleshooting: The most common cause of a sudden spike in log volumes is generally not sudden increased  load on a system, but unfortunately is more often a result of a misconfiguration or application bug that starts to produce cascading errors. A sudden spike from a specific log is often an indicator of issues emanating from a particular application component. Similarly if logs suddenly become inactive, it may signify something has gone horribly wrong.
  • Infrastructure usage analysis: When your cloud bills start to grow, it is common for CFOs and IT leads to sit down weekly/monthly to understand what is driving this. This has been notoriously difficult in the past and has resulted in the rise of companies like cloudability and cloudtechhealth for example. Looking at your cloud costs breakdown along side log level usage data can help you correlate and understand some of the drivers behind these costs, especially if there has been significant change in your cloud cost structure.

To sign up for your weekly log level usage report simply go here and provide your details.


Sign-up for a free 30 Day Trial

Get set up in minutes, and gain insights in seconds

Start Free Trial Setup a Demo

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.

Latest Stories
In the 21st century, security on the Internet has become one of the most important issues. We hear more and more about cyber-attacks on the websites of large corporations, banks and even small businesses. When online we’re concerned not only for our own safety but also our privacy. We have to know that hackers usually start their preparation by investigating the private information of admins – the habits, interests, visited websites and so on. On the other hand, our own security is in danger bec...
The Internet of Things (IoT), in all its myriad manifestations, has great potential. Much of that potential comes from the evolving data management and analytic (DMA) technologies and processes that allow us to gain insight from all of the IoT data that can be generated and gathered. This potential may never be met as those data sets are tied to specific industry verticals and single markets, with no clear way to use IoT data and sensor analytics to fulfill the hype being given the IoT today.
Enterprises have been using both Big Data and virtualization for years. Until recently, however, most enterprises have not combined the two. Big Data's demands for higher levels of performance, the ability to control quality-of-service (QoS), and the ability to adhere to SLAs have kept it on bare metal, apart from the modern data center cloud. With recent technology innovations, we've seen the advantages of bare metal erode to such a degree that the enhanced flexibility and reduced costs that cl...
Without lifecycle traceability and visibility across the tool chain, stakeholders from Planning-to-Ops have limited insight and answers to who, what, when, why and how across the DevOps lifecycle. This impacts the ability to deliver high quality software at the needed velocity to drive positive business outcomes. In his session at @DevOpsSummit 19th Cloud Expo, Eric Robertson, General Manager at CollabNet, will show how customers are able to achieve a level of transparency that enables everyon...
Donna Yasay, President of HomeGrid Forum, today discussed with a panel of technology peers how certification programs are at the forefront of interoperability, and the answer for vendors looking to keep up with today's growing industry for smart home innovation. "To ensure multi-vendor interoperability, accredited industry certification programs should be used for every product to provide credibility and quality assurance for retail and carrier based customers looking to add ever increasing num...
“Media Sponsor” of SYS-CON's 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. CloudBerry Backup is a leading cross-platform cloud backup and disaster recovery solution integrated with major public cloud services, such as Amazon Web Services, Microsoft Azure and Google Cloud Platform.
In the next forty months – just over three years – businesses will undergo extraordinary changes. The exponential growth of digitization and machine learning will see a step function change in how businesses create value, satisfy customers, and outperform their competition. In the next forty months companies will take the actions that will see them get to the next level of the game called Capitalism. Or they won’t – game over. The winners of today and tomorrow think differently, follow different...
@DevOpsSummit has been named the ‘Top DevOps Influencer' by iTrend. iTrend processes millions of conversations, tweets, interactions, news articles, press releases, blog posts - and extract meaning form them and analyzes mobile and desktop software platforms used to communicate, various metadata (such as geo location), and automation tools. In overall placement, @DevOpsSummit ranked as the number one ‘DevOps Influencer' followed by @CloudExpo at third, and @MicroservicesE at 24th.
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, will discuss how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team a...
The security needs of IoT environments require a strong, proven approach to maintain security, trust and privacy in their ecosystem. Assurance and protection of device identity, secure data encryption and authentication are the key security challenges organizations are trying to address when integrating IoT devices. This holds true for IoT applications in a wide range of industries, for example, healthcare, consumer devices, and manufacturing. In his session at @ThingsExpo, Lancen LaChance, vic...
Regulatory requirements exist to promote the controlled sharing of information, while protecting the privacy and/or security of the information. Regulations for each type of information have their own set of rules, policies, and guidelines. Cloud Service Providers (CSP) are faced with increasing demand for services at decreasing prices. Demonstrating and maintaining compliance with regulations is a nontrivial task and doing so against numerous sets of regulatory requirements can be daunting task...
What are the successful IoT innovations from emerging markets? What are the unique challenges and opportunities from these markets? How did the constraints in connectivity among others lead to groundbreaking insights? In her session at @ThingsExpo, Carmen Feliciano, a Principal at AMDG, will answer all these questions and share how you can apply IoT best practices and frameworks from the emerging markets to your own business.
Between the mockups and specs produced by analysts, and resulting applications built by developers, there exists a gulf where projects fail, costs spiral, and applications disappoint. Methodologies like Agile attempt to address this with intensified communication, with partial success but many limitations. In his session at @DevOpsSummit at 19th Cloud Expo, Charles Kendrick, CTO at Isomorphic Software, will present a revolutionary model enabled by new technologies. Learn how business and deve...
Big Data has been changing the world. IoT fuels the further transformation recently. How are Big Data and IoT related? In his session at @BigDataExpo, Tony Shan, a renowned visionary and thought leader, will explore the interplay of Big Data and IoT. He will anatomize Big Data and IoT separately in terms of what, which, why, where, when, who, how and how much. He will then analyze the relationship between IoT and Big Data, specifically the drilldown of how the 4Vs of Big Data (Volume, Variety,...
SYS-CON Events announced today that SoftNet Solutions will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. SoftNet Solutions specializes in Enterprise Solutions for Hadoop and Big Data. It offers customers the most open, robust, and value-conscious portfolio of solutions, services, and tools for the shortest route to success with Big Data. The unique differentiator is the ability to architect and ...