Blog Feed Post

From Scala Unified Logging to Full System ObservabilityPart 2 of 3: How We Made Logging Great Again

Jonathan is a platform engineer at VictorOps, responsible for system scalability and performance. This is the second part in a series on system visibility, the Detection and Analysis part of the Incident Management Lifecycle. If you missed it, read Part 1 first.

Unified Interface

Like any good engineering approach, we wanted a simple, thin facade that would codify our use-cases, provide a single extension point, and be low-overhead to maintain. With the exception of appender performance, we needed unification of the interfaces in order to address all the concerns mentioned in Part 1. We did just that.

We created a top level Logger that would house all supplemental functionality and become our primary interface regardless of whether we’re logging from a “typical” class, an actor, or a controller. There are two implementations: ClassLogger and ActorLogger which behind the scenes make use of SLF4J and Akka logging, respectively. So far, we haven’t had a use case for a separate controller-oriented logger and simply used the ClassLogger in our controllers.

Next, we created a Logging trait in order to further standardize our logger names and break the stigma of a separate Logger for each class in a class hierarchy, in favor of a single Logger per instance. In other words, we lose the context of which hierarchical class in favor of the gains it brings. This has the benefit of providing better context to any log statements called from a parent class or trait mixin. This also removed further developer decisions by mixing in a logger with the same field name, logger name, and identity* which we’ll get into next, regarding log variables.

*identity–in this case I’m referring to the information you want printed in every log statement from that class. For example, if the class is associated to a specific user, the username may be a part of its identity.

Our supplemental functionality

Support for Log Variables

The goal of codifying use-cases was achieved through adding first class support for log variables — which we define as any key-value pair you want easily parsed from your log statement. This meant the developer was now able to simply pass a tuple (or with macros, just the aptly named variable) to the log method and it was formatted by the facade. No more formatting errors, like this:

Notice the missing closing brace and, in this case, the inconsistency of this statement using colon+space instead of equals. That simple mistake would lead to a failure to extract the variable in your log parsing (in Sumo/Elastic/Splunk/etc) making it somewhat hidden and potentially yield the log statement useless.

Once we had basic support for log variables we got some ideas. For example, how about allowing a class to set class-level log variables that are printed in every log statement from a class? We did just that. Then, we added the ability to simply create a formatted string using the same functionality – useful for strings given to other objects, like a failed Future. Below, see how we were able to expand the use of class-level log variables as well as string creation into failure messages:

Implicit loggers for utility classes

Another perspective on log variables is that they allow you to capture the context of a log statement which helps to describe the data being acted upon. Implicit loggers allow utility objects to assume the identity of the caller when logging relevant information about their job. So, instead of telling you about a needle in a haystack:

We now specify who wanted it (since the class represents the caller) as well as any context they set up in their logger (like the org and user in this example):

Sanitizing PII

A feature worth considering is sanitizing PII (personally identifiable information) in log statements. There are some references to functionality like this in the GoDaddy logger and others. So, focusing on maintaining a thin facade, you could completely replace Logback with the GoDaddy Logger and add this functionality without modifying many, if any, log statements directly.

Is Logging Great Again?

This simple solution totaled 452 lines of code, including tests and nearly 100 lines of comments to aid in migrating files — the developer responses were surprising to say the least.

Remember the last time you got a splinter in your foot? Such a simple task of removing the splinter relieves that constant annoyance that can easily build up if not addressed. Ha ha! Our state of logging was just like that splinter. Relieved comments began to trickle in, one by one, on how devs have been pleasantly surprised with the new experience of logging on our platform.

Most of those comments centered on how the unified logging was implemented at the right level of abstraction (flexibility versus structure) and that it made logging simple and, because of that, enjoyable to use. These days at VictorOps, developers are progressively migrating the files they touch in order to reap the benefits and the simplicity they’ve become accustomed to.

Here’s a before/after displaying the reuse, codified formatting, and maintainability improvements:

With this change, we progressed in two very important ways as a team: We addressed some tech debt that had become a larger nuisance than we had realized; and, we improved our log searchability by providing simplicity and just enough structure to get the job done.

So, with our logging in a much happier place, we were prepared to move to the next step of improving our monitoring and instrumentation. In Part 3, we’ll talk about how metrics have become our next focal point.

The post From Scala Unified Logging to Full System Observability
Part 2 of 3: How We Made Logging Great Again
appeared first on VictorOps.

Read the original blog entry...

More Stories By VictorOps Blog

VictorOps is making on-call suck less with the only collaborative alert management platform on the market.

With easy on-call scheduling management, a real-time incident timeline that gives you contextual relevance around your alerts and powerful reporting features that make post-mortems more effective, VictorOps helps your IT/DevOps team solve problems faster.

Latest Stories
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
In their session at @ThingsExpo, Shyam Varan Nath, Principal Architect at GE, and Ibrahim Gokcen, who leads GE's advanced IoT analytics, focused on the Internet of Things / Industrial Internet and how to make it operational for business end-users. Learn about the challenges posed by machine and sensor data and how to marry it with enterprise data. They also discussed the tips and tricks to provide the Industrial Internet as an end-user consumable service using Big Data Analytics and Industrial C...
René Bostic is the Technical VP of the IBM Cloud Unit in North America. Enjoying her career with IBM during the modern millennial technological era, she is an expert in cloud computing, DevOps and emerging cloud technologies such as Blockchain. Her strengths and core competencies include a proven record of accomplishments in consensus building at all levels to assess, plan, and implement enterprise and cloud computing solutions. René is a member of the Society of Women Engineers (SWE) and a m...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term.
To Really Work for Enterprises, MultiCloud Adoption Requires Far Better and Inclusive Cloud Monitoring and Cost Management … But How? Overwhelmingly, even as enterprises have adopted cloud computing and are expanding to multi-cloud computing, IT leaders remain concerned about how to monitor, manage and control costs across hybrid and multi-cloud deployments. It’s clear that traditional IT monitoring and management approaches, designed after all for on-premises data centers, are falling short in ...
With privacy often voiced as the primary concern when using cloud based services, SyncriBox was designed to ensure that the software remains completely under the customer's control. Having both the source and destination files remain under the user?s control, there are no privacy or security issues. Since files are synchronized using Syncrify Server, no third party ever sees these files.
Mobile device usage has increased exponentially during the past several years, as consumers rely on handhelds for everything from news and weather to banking and purchases. What can we expect in the next few years? The way in which we interact with our devices will fundamentally change, as businesses leverage Artificial Intelligence. We already see this taking shape as businesses leverage AI for cost savings and customer responsiveness. This trend will continue, as AI is used for more sophistica...
"We are an integrator of carrier ethernet and bandwidth to get people to connect to the cloud, to the SaaS providers, and the IaaS providers all on ethernet," explained Paul Mako, CEO & CTO of Massive Networks, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
I believe that this may finally be the year that the CIO role ‘crosses the Rubicon,' leaving behind its traditional, IT-focused orientation. But I don't believe that either of the previous predictions of this outcome — fading into oblivion or rising to a business executive level — is correct. Instead, I think this is the year that we will see the role of the CIO transformed into something altogether different.
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
"Calligo is a cloud service provider with data privacy at the heart of what we do. We are a typical Infrastructure as a Service cloud provider but it's been designed around data privacy," explained Julian Box, CEO and co-founder of Calligo, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"NetApp is known as a data management leader but we do a lot more than just data management on-prem with the data centers of our customers. We're also big in the hybrid cloud," explained Wes Talbert, Principal Architect at NetApp, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.