Welcome!

Blog Feed Post

From Scala Unified Logging to Full System ObservabilityPart 2 of 3: How We Made Logging Great Again

Jonathan is a platform engineer at VictorOps, responsible for system scalability and performance. This is the second part in a series on system visibility, the Detection and Analysis part of the Incident Management Lifecycle. If you missed it, read Part 1 first.

Unified Interface

Like any good engineering approach, we wanted a simple, thin facade that would codify our use-cases, provide a single extension point, and be low-overhead to maintain. With the exception of appender performance, we needed unification of the interfaces in order to address all the concerns mentioned in Part 1. We did just that.

We created a top level Logger that would house all supplemental functionality and become our primary interface regardless of whether we’re logging from a “typical” class, an actor, or a controller. There are two implementations: ClassLogger and ActorLogger which behind the scenes make use of SLF4J and Akka logging, respectively. So far, we haven’t had a use case for a separate controller-oriented logger and simply used the ClassLogger in our controllers.


Next, we created a Logging trait in order to further standardize our logger names and break the stigma of a separate Logger for each class in a class hierarchy, in favor of a single Logger per instance. In other words, we lose the context of which hierarchical class in favor of the gains it brings. This has the benefit of providing better context to any log statements called from a parent class or trait mixin. This also removed further developer decisions by mixing in a logger with the same field name, logger name, and identity* which we’ll get into next, regarding log variables.

*identity–in this case I’m referring to the information you want printed in every log statement from that class. For example, if the class is associated to a specific user, the username may be a part of its identity.

Our supplemental functionality

Support for Log Variables

The goal of codifying use-cases was achieved through adding first class support for log variables — which we define as any key-value pair you want easily parsed from your log statement. This meant the developer was now able to simply pass a tuple (or with macros, just the aptly named variable) to the log method and it was formatted by the facade. No more formatting errors, like this:

Notice the missing closing brace and, in this case, the inconsistency of this statement using colon+space instead of equals. That simple mistake would lead to a failure to extract the variable in your log parsing (in Sumo/Elastic/Splunk/etc) making it somewhat hidden and potentially yield the log statement useless.

Once we had basic support for log variables we got some ideas. For example, how about allowing a class to set class-level log variables that are printed in every log statement from a class? We did just that. Then, we added the ability to simply create a formatted string using the same functionality – useful for strings given to other objects, like a failed Future. Below, see how we were able to expand the use of class-level log variables as well as string creation into failure messages:

Implicit loggers for utility classes


Another perspective on log variables is that they allow you to capture the context of a log statement which helps to describe the data being acted upon. Implicit loggers allow utility objects to assume the identity of the caller when logging relevant information about their job. So, instead of telling you about a needle in a haystack:

We now specify who wanted it (since the class represents the caller) as well as any context they set up in their logger (like the org and user in this example):

Sanitizing PII

A feature worth considering is sanitizing PII (personally identifiable information) in log statements. There are some references to functionality like this in the GoDaddy logger and others. So, focusing on maintaining a thin facade, you could completely replace Logback with the GoDaddy Logger and add this functionality without modifying many, if any, log statements directly.

Is Logging Great Again?

This simple solution totaled 452 lines of code, including tests and nearly 100 lines of comments to aid in migrating files — the developer responses were surprising to say the least.

Remember the last time you got a splinter in your foot? Such a simple task of removing the splinter relieves that constant annoyance that can easily build up if not addressed. Ha ha! Our state of logging was just like that splinter. Relieved comments began to trickle in, one by one, on how devs have been pleasantly surprised with the new experience of logging on our platform.

Most of those comments centered on how the unified logging was implemented at the right level of abstraction (flexibility versus structure) and that it made logging simple and, because of that, enjoyable to use. These days at VictorOps, developers are progressively migrating the files they touch in order to reap the benefits and the simplicity they’ve become accustomed to.

Here’s a before/after displaying the reuse, codified formatting, and maintainability improvements:

With this change, we progressed in two very important ways as a team: We addressed some tech debt that had become a larger nuisance than we had realized; and, we improved our log searchability by providing simplicity and just enough structure to get the job done.

So, with our logging in a much happier place, we were prepared to move to the next step of improving our monitoring and instrumentation. In Part 3, we’ll talk about how metrics have become our next focal point.

The post From Scala Unified Logging to Full System Observability
Part 2 of 3: How We Made Logging Great Again
appeared first on VictorOps.

Read the original blog entry...

More Stories By VictorOps Blog

VictorOps is making on-call suck less with the only collaborative alert management platform on the market.

With easy on-call scheduling management, a real-time incident timeline that gives you contextual relevance around your alerts and powerful reporting features that make post-mortems more effective, VictorOps helps your IT/DevOps team solve problems faster.

Latest Stories
This session will provide an introduction to Cloud driven quality and transformation and highlight the key features that comprise it. A perspective on the cloud transformation lifecycle, transformation levers, and transformation framework will be shared. At Cognizant, we have developed a transformation strategy to enable the migration of business critical workloads to cloud environments. The strategy encompasses a set of transformation levers across the cloud transformation lifecycle to enhance ...
Your job is mostly boring. Many of the IT operations tasks you perform on a day-to-day basis are repetitive and dull. Utilizing automation can improve your work life, automating away the drudgery and embracing the passion for technology that got you started in the first place. In this presentation, I'll talk about what automation is, and how to approach implementing it in the context of IT Operations. Ned will discuss keys to success in the long term and include practical real-world examples. Ge...
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
So the dumpster is on fire. Again. The site's down. Your boss's face is an ever-deepening purple. And you begin debating whether you should join the #incident channel or call an ambulance to deal with his impending stroke. Yes, we know this is a developer's fault. There's plenty of time for blame later. Postmortems have a macabre name because they were once intended to be Viking-like funerals for someone's job. But we're civilized now. Sort of. So we call them post-incident reviews. Fires are ne...
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments t...
Hackers took three days to identify and exploit a known vulnerability in Equifax’s web applications. I will share new data that reveals why three days (at most) is the new normal for DevSecOps teams to move new business /security requirements from design into production. This session aims to enlighten DevOps teams, security and development professionals by sharing results from the 4th annual State of the Software Supply Chain Report -- a blend of public and proprietary data with expert researc...
CloudEXPO New York 2018, colocated with DevOpsSUMMIT and DXWorldEXPO New York 2018 will be held November 12-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI and Machine Learning to one location.
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
DXWorldEXPO LLC announced today that Nutanix has been named "Platinum Sponsor" of CloudEXPO | DevOpsSUMMIT | DXWorldEXPO New York, which will take place November 12-13, 2018 in New York City. Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that power their business. The Nutanix Enterprise Cloud Platform blends web-scale engineering and consumer-grade design to natively converge server, storage, virtualization and networking into a resilient, softwar...
The digital transformation is real! To adapt, IT professionals need to transform their own skillset to become more multi-dimensional by gaining both depth and breadth of a wide variety of knowledge and competencies. Historically, while IT has been built on a foundation of specialty (or "I" shaped) silos, the DevOps principle of "shifting left" is opening up opportunities for developers, operational staff, security and others to grow their skills portfolio, advance their careers and become "T"-sh...
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app secu...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...
ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of computational needs for many industries. Their solutions provide benefits across many environments, such as datacenter deployment, HPC, workstations, storage networks and standalone server installations. ICC has been in business for over 23 years and their phenomenal range of clients include multinational corporations, universities, and small busines...
This sixteen (16) hour course provides an introduction to DevOps, the cultural and professional movement that stresses communication, collaboration, integration and automation in order to improve the flow of work between software developers and IT operations professionals. Improved workflows will result in an improved ability to design, develop, deploy and operate software and services faster.
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...