Welcome!

Blog Feed Post

What is structured logging and why developers need it

Log files are one of the most valuable assets that a developer has. Usually when something goes wrong in production the first thing you hear is “send me the logs”. The goal of structured logging is to bring a more defined format and details to your logging. We have been practicing structured logging at Stackify for quite a while and want to share some of our thoughts and best practices.

What is structured logging?

The problem with log files is they are unstructured text data. This makes it hard to query them for any sort of useful information. As a developer, it would be nice to be able to filter all logs by a certain customer # or transaction #. The goal of structured logging is to solve these sorts of problems and allow additional analytics.

For log files to be machine readable more advanced functionality, they need to be written in a structured format that can easily parsed. This could be XML, JSON, or other formats. But since virtually everything these days is JSON, you are most likely to see JSON as the standard format for structured logging.

Structured logging can be used for a couple different use cases:

  1. Process log files for analytics or business intelligence – A good example of this would be processing web server access logs and doing some basic summarization and aggregates across the data.
  2. Searching log files – Being able to search and correlate log messages is very valuable to development teams during the development process and for troubleshooting production problems.

Structured logging example

A simple example will probably help to make it clear as to what structured logging really is.

Normally you might write to a log file like this:

log.Debug("Incoming metrics data");

This would produce a line like this in your log:

DEBUG 2017-01-27 16:17:58 – Incoming metrics data

Depending on your logging framework, logging some additional fields would be done like this. This give you the ability to potentially easily search on these custom fields.

log.Debug("Incoming metrics data", new {clientid=54732});

This would produce a line like this in your log, now including the extra field:

DEBUG 2017-01-27 16:17:58 – Incoming metrics data {"clientid":54732}

If you were using structured logging and sending it to a log management system, it would serialize the entire message and additional metadata as JSON. This is part of the power of using structured logs and a log management system that supports them.

[{
		"Env" : "Unknown",
		"ServerName" : "LAPTOP1",
		"AppName" : "ConsoleApplication1.vshost.exe",
		"AppLoc" : "C:\\BitBucket\\stackify-api-dotnet\\Src\\ConsoleApplication1\\bin\\Debug\\ConsoleApplication1.vshost.exe",
		"Logger" : "StackifyLib.net",
		"Platform" : ".net",
		"Msgs" : [{
				"Msg" : "Incoming metrics data",
				"data" : "{\"clientid\":54732}",
				"Thread" : "10",
				"EpochMs" : 1485555302470,
				"Level" : "DEBUG",
				"id" : "0c28701b-e4de-11e6-8936-8975598968a4"
			}
		]
	}
]

It is important to know that there is no real standard to structured logging and it can be done a lot of different ways. To get the most value out of it, you need to be using a logging framework (like log4net, log4j, etc) that supports logging additional properties and then send that data to a log management system that can accept your custom fields and index them.

How to view structured logs

If you are programming with .NET or Java, you can use Prefix to view what your code is doing via transaction tracing along with your logging. Prefix can even show you any custom properties that are being logged as JSON. Prefix is free and is the best log viewer developers can get.

prefix-error-log-full-trace-2

How we use structured logging at Stackify

At Stackify we use structured logging primarily to make it easier to search our logs. We care more about the benefits it provides for our developers.

When we look at our logs, they look like this below. You can see all the custom fields we log because they show up as JSON.

structured logging demo

This enables us to very easily search by any of those fields via our log management system.

A simple search like this: “clientidNumber:54732”, shows us only those logs, helping us quickly narrow down problems for a specific client. I can search across every app and server we have from one place.

structured logging demo filtered

TIP: Log extra fields on exceptions!

One of the best uses of structured logging is on exceptions. Trying to figure out why an exception happened is infinitely easier if you know more details about who the user was, input parameters, etc.

    try
    {
        //do something
    }
    catch (Exception ex)
    {
        log.Error("Error trying to do something", new { clientid = 54732, user = "matt" }, ex);
    }

Final thoughts on structured logging

It doesn’t really take any longer to log custom properties as you write your logging. These extra properties can provide more details that make it easier to trouble application problems. If you are using a log management system that supports searching by these custom fields, then you can also search your logs by these new properties.

If you need help with structured logging, be sure to try out Retrace which includes logging for free as a standard feature.

The post What is structured logging and why developers need it appeared first on Stackify.

Read the original blog entry...

More Stories By Stackify Blog

Stackify offers the only developers-friendly solution that fully integrates error and log management with application performance monitoring and management. Allowing you to easily isolate issues, identify what needs to be fixed quicker and focus your efforts – Support less, Code more. Stackify provides software developers, operations and support managers with an innovative cloud based solution that gives them DevOps insight and allows them to monitor, detect and resolve application issues before they affect the business to ensure a better end user experience. Start your free trial now stackify.com

Latest Stories
SYS-CON Events announced today that Linux Academy, the foremost online Linux and cloud training platform and community, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Linux Academy was founded on the belief that providing high-quality, in-depth training should be available at an affordable price. Industry leaders in quality training, provided services, and student certification passes, its goal is to c...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In his Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, will explore t...
Providing the needed data for application development and testing is a huge headache for most organizations. The problems are often the same across companies - speed, quality, cost, and control. Provisioning data can take days or weeks, every time a refresh is required. Using dummy data leads to quality problems. Creating physical copies of large data sets and sending them to distributed teams of developers eats up expensive storage and bandwidth resources. And, all of these copies proliferating...
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
SYS-CON Events announced today that StorageCraft Technology Corp, a global leader in backup and disaster, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. The StorageCraft family of companies, founded in 2003, provides award-winning backup, disaster recovery, system migration and data protection solutions for servers, desktops and laptops in addition to powerful data analytics.
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), will provide an overview of various initiatives to certifiy the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldw...
SYS-CON Events announced today that Auditwerx will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Auditwerx specializes in SOC 1, SOC 2, and SOC 3 attestation services throughout the U.S. and Canada. As a division of Carr, Riggs & Ingram (CRI), one of the top 20 largest CPA firms nationally, you can expect the resources, skills, and experience of a much larger firm combined with the accessibility and attent...
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ pr...
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
HyperConvergence came to market with the objective of being simple, flexible and to help drive down operating expenses. It reduced the footprint by bundling the compute/storage/network into one box. This brought a new set of challenges as the HyperConverged vendors are very focused on their own proprietary building blocks. If you want to scale in a certain way, let’s say you identified a need for more storage and want to add a device that is not sold by the HyperConverged vendor, forget about it...
FinTech is the sum of financial and technology, and it’s one of the fastest growing tech industries. Total global investments in FinTech almost reached $50 billion last year, but there is still a great deal of confusion over what it is and what it means – especially as it applies to retirement. Building financial startups is not simple, but with the right team, technology and an innovative approach it can be an extremely interesting domain to disrupt. FinTech heralds a financial revolution that...
What if you could build a web application that could support true web-scale traffic without having to ever provision or manage a single server? Sounds magical, and it is! In his session at 20th Cloud Expo, Chris Munns, Senior Developer Advocate for Serverless Applications at Amazon Web Services, will show how to build a serverless website that scales automatically using services like AWS Lambda, Amazon API Gateway, and Amazon S3. We will review several frameworks that can help you build serverle...
Most companies are adopting or evaluating container technology - Docker in particular - to speed up application deployment, drive down cost, ease management and make application delivery more flexible overall. As with most new architectures, this dream takes a lot of work to become a reality. Even when you do get your application componentized enough and packaged properly, there are still challenges for DevOps teams to making the shift to continuous delivery and achieving that reduction in cost ...
SYS-CON Events announced today that Loom Systems will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2015, Loom Systems delivers an advanced AI solution to predict and prevent problems in the digital business. Loom stands alone in the industry as an AI analysis platform requiring no prior math knowledge from operators, leveraging the existing staff to succeed in the digital era. With offices in S...