Blog Feed Post

Introducing fully automated support for Go-based application monitoring

We’re proud to announce the immediate beta availability of Dynatrace OneAgent for Go, the open-source programming language. This is the industry’s first auto-instrumenting monitoring solution available for Go-based applications. No code changes to your Go applications are required to use Dynatrace. As such, OneAgent for Go can monitor not only your own Go applications, it can also monitor 3rd party Go-based applications that your applications may rely on.

The unique OneAgent for Go monitoring technology enables you to capture data that is well beyond the capabilities of traditional monitoring solutions. Dynatrace can extract internal Go runtime information that is inaccessible with public Go runtime APIs.

Activate Go support

Monitoring of Go applications is disabled by default during the beta release period.

Note: This feature requires Dynatrace OneAgent version 125 or higher and Dynatrace Cluster version 127 or higher.

To activate Go support

  1. Go to Settings > Monitoring > Monitored technologies.
  2. Set the Go switch to the On position.

Monitor Go with Dynatrace

To illustrate how Go monitoring works, and to show you the available metrics that Dynatrace provides for Go applications, we’ll use an example influxdb database that receives requests via a load-test script. This example uses the out-of-the-box influxdb Ubuntu package binary. To begin monitoring, following installation of Dynatrace OneAgent and activation of Go support (see above), restart the influxdb service by running the command systemctl restart influxdb.

After the restart the influxd process front page

Following restart of the influxd process, the process view will automatically display Go specific metrics  (see example below). Note that the Go version that the executable was built with (1.7.4)  is listed under Properties.

Key metrics

Dynatrace OneAgent immediately begins tracking the following important Go metrics.

Suspension Go garbage collector percentage share compared to overall application CPU time.
Runtime heap The number of bytes used/committed to Go heap and Go garbage collector execution times.
Goroutine count The number of Go routines instantiated by the application and Go runtime infrastructure.

Click the Further details tab to drill down to the details of the running process.

Go managed memory

The Go managed memory tab breaks down memory metrics into various categories:

Heap Bytes of memory used/committed to Go runtime heap.
Stack Bytes of memory used/committed to dynamic Go stacks. Go stacks are used to execute Go routines and grow dynamically.
OffHeap Bytes of memory used/committed for Go runtime internal structures that are not allocated from heap memory. Data structures used in Go heap implementation are an example of OffHeap memory.
Overall The sum of Heap, OffHeap, and Stack memory.

Heap details

The Heap details tab digs deeper into the anatomy of Go heaps.

Heap allocated Go objects count Number of Go objects allocated on the heap.
Heap idle size Number of bytes currently not assigned to Go heap or stack. Idle memory may be returned to OS, or retained by Go runtime for later reassignment to heap or stack category.
Heep live size Number of bytes considered live by the Go garbage collector. This metric accumulates memory retained by the most recent garbage collector run and allocated since then.
Garbage collector invocation count Number of Go garbage collector runs.

Understand Goroutine scheduling

Go does a fantastic job of hiding Goroutine scheduling details from application developers. A basic understanding of the internal scheduling mechanics will help you read scheduling metrics and detect potential anomalies. As documentation of Go internals is sparse, following is a short primer to get you started.

The implementation of Goroutine scheduling deals with three central object types: M (Machine), P (Processor), and G (Goroutine). You’ll find many references to these object types by browsing through Go runtime source code. For expressiveness, we use the following alternative terms for M, P, and G:

Go runtime nomenclature Expressive name
M Worker thread
P Scheduling context
G Goroutine

Go executes Goroutines in context of worker threads acquired from a pool of native operating system threads. A Goroutine may be assigned to a different worker thread at any point in time (unless runtime.LockOSThread is used to enforce worker thread affinity).

Multiple Goroutines are typically assigned to a single worker thread. A scheduling context is responsible for the cooperative scheduling of these Goroutines. The Go compiler adds code to each Go function prologue which checks if the currently executing Goroutine consumed its 10 milliseconds execution time slice (the actual mechanism cleverly uses stack guards to enforce rescheduling). If the time slice is exceeded, the scheduling context sets up the next Goroutine to execute. This is why scheduling is cooperative: if a Goroutine doesn’t invoke Go functions, it won’t be rescheduled.

Each set of Goroutines is executed by a worker thread. Execution is controlled by a scheduling context. But what happens if a Goroutine writes a large chunk of data to disk or blocks waiting for an incoming connection? The Goroutine will be blocked in a system call and no other Goroutine will be scheduled (remember, reschedule happens only upon invocation of Go functions). Thus all Goroutines assigned to the same worker thread will also be blocked!

But no worries, Go deals elegantly with this situation. If a Goroutine executes a blocking system call, the scheduling context with the other Goroutines will be assigned on the fly to a different worker thread (either a parked or newly instantiated thread). This explains why the total number of worker threads is larger than the number of scheduling context objects. Once the blocking call returns, the Goroutine is again assigned to a scheduling context or, if the assignment fails, to the global Goroutine run queue. By the way, the very same principle applies to cgo (Go to C language) calls.

It’s notable that the number of scheduling contexts is the only user configurable setting in the Go scheduling algorithm (see the GOMAXPROCS environment variable and runtime.GOMAXPROCS function). You can’t control the number of worker threads.

Therefore, writing a large chunk of data to disk or waiting for incoming connections won’t block other Goroutines that are initially assigned to the same worker thread. These Goroutines will continue execution on their newly assigned worker threads.

With these fundamentals in mind, you’re now ready to dive into scheduling metrics.


The Scheduling tab provides unique insights into Goroutine scheduling.

System call count Number of system calls executed by Go runtime. The number does not include system calls performed in the context of cgo.
cgo call count Number of cgo calls.
Parked worker threads count Number of worker threads parked by Go runtime. A parked worker thread doesn’t consume CPU cycles until Go runtime unparks the thread.
“Out of work” worker threads count A worker thread is considered out of work when the associated scheduling context has no more Goroutines to execute. The worker thread attempts to steal Goroutines from another scheduling context and global run queue. If the stealing fails, the worker thread will park itself after some time.

The same mechanism applies to a high work load scenario. If an idle scheduling context exists, Go runtime will “unpark” a parked worker thread and associate the idle scheduling context. The unparked worker thread is now in the ‘out of work’ state and will start Goroutine stealing.

Worker thread count Number of operating system threads instantiated to execute Go routines. Go does not terminate worker threads; it keeps them in a parked state for future re-use.
Global Goroutine run queue size Number of Goroutines in the global run queue. Goroutines are placed in the global run queue if the worker thread used to execute a blocking system call can’t acquire a scheduling context. Scheduling contexts periodically acquire Goroutines from the global run queue.
Idle scheduling context count A scheduling context is considered idle if it has no more Goroutines to execute and Goroutine acquisition from the global run queue or other scheduling contexts have failed.

Future outlook

This initial beta release of Dynatrace Go monitoring provides some great metrics and Go version detection, but there’s much more to come! The next version of Dynatrace Go monitoring will feature full web-request tracing capabilities—similar to those currently offered for all other supported technologies. Our goal is to make Dynatrace the first and only full-stack monitoring solution in the world that supports end-to-end transaction monitoring of Go processes, with no required changes to your code. As Go adoption grows, this promises to be a game changer.


Many Dynatrace folks have made important contributions and valuable input to Dynatrace Go technology support. But one individual deserves special acknowledgment: Michael “Mr. Runtime” Obermueller. Our Go support would not have been possible without Michael’s valuable input and vast knowledge of Go internal mechanisms.

The post Introducing fully automated support for Go-based application monitoring appeared first on Dynatrace blog – monitoring redefined.

Read the original blog entry...

More Stories By Dynatrace Blog

Building a revolutionary approach to software performance monitoring takes an extraordinary team. With decades of combined experience and an impressive history of disruptive innovation, that’s exactly what we ruxit has.

Get to know ruxit, and get to know the future of data analytics.

Latest Stories
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
So the dumpster is on fire. Again. The site's down. Your boss's face is an ever-deepening purple. And you begin debating whether you should join the #incident channel or call an ambulance to deal with his impending stroke. Yes, we know this is a developer's fault. There's plenty of time for blame later. Postmortems have a macabre name because they were once intended to be Viking-like funerals for someone's job. But we're civilized now. Sort of. So we call them post-incident reviews. Fires are ne...
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments t...
Hackers took three days to identify and exploit a known vulnerability in Equifax’s web applications. I will share new data that reveals why three days (at most) is the new normal for DevSecOps teams to move new business /security requirements from design into production. This session aims to enlighten DevOps teams, security and development professionals by sharing results from the 4th annual State of the Software Supply Chain Report -- a blend of public and proprietary data with expert researc...
CloudEXPO New York 2018, colocated with DevOpsSUMMIT and DXWorldEXPO New York 2018 will be held November 12-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI and Machine Learning to one location.
DXWorldEXPO LLC announced today that Nutanix has been named "Platinum Sponsor" of CloudEXPO | DevOpsSUMMIT | DXWorldEXPO New York, which will take place November 12-13, 2018 in New York City. Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that power their business. The Nutanix Enterprise Cloud Platform blends web-scale engineering and consumer-grade design to natively converge server, storage, virtualization and networking into a resilient, softwar...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
The digital transformation is real! To adapt, IT professionals need to transform their own skillset to become more multi-dimensional by gaining both depth and breadth of a wide variety of knowledge and competencies. Historically, while IT has been built on a foundation of specialty (or "I" shaped) silos, the DevOps principle of "shifting left" is opening up opportunities for developers, operational staff, security and others to grow their skills portfolio, advance their careers and become "T"-sh...
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app secu...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...
ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of computational needs for many industries. Their solutions provide benefits across many environments, such as datacenter deployment, HPC, workstations, storage networks and standalone server installations. ICC has been in business for over 23 years and their phenomenal range of clients include multinational corporations, universities, and small busines...
This sixteen (16) hour course provides an introduction to DevOps, the cultural and professional movement that stresses communication, collaboration, integration and automation in order to improve the flow of work between software developers and IT operations professionals. Improved workflows will result in an improved ability to design, develop, deploy and operate software and services faster.
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
All in Mobile is a place where we continually maximize their impact by fostering understanding, empathy, insights, creativity and joy. They believe that a truly useful and desirable mobile app doesn't need the brightest idea or the most advanced technology. A great product begins with understanding people. It's easy to think that customers will love your app, but can you justify it? They make sure your final app is something that users truly want and need. The only way to do this is by ...
Authorization of web applications developed in the cloud is a fundamental problem for security, yet companies often build solutions from scratch, which is error prone and impedes time to market. This talk shows developers how they can (instead) build on-top of community-owned projects and frameworks for better security.Whether you build software for enterprises, mobile, or internal microservices, security is important. Standards like SAML, OIDC, and SPIFFE help you solve identity and authenticat...