Blog Feed Post

Introducing fully automated support for Go-based application monitoring

We’re proud to announce the immediate beta availability of Dynatrace OneAgent for Go, the open-source programming language. This is the industry’s first auto-instrumenting monitoring solution available for Go-based applications. No code changes to your Go applications are required to use Dynatrace. As such, OneAgent for Go can monitor not only your own Go applications, it can also monitor 3rd party Go-based applications that your applications may rely on.

The unique OneAgent for Go monitoring technology enables you to capture data that is well beyond the capabilities of traditional monitoring solutions. Dynatrace can extract internal Go runtime information that is inaccessible with public Go runtime APIs.

Activate Go support

Monitoring of Go applications is disabled by default during the beta release period.

Note: This feature requires Dynatrace OneAgent version 125 or higher and Dynatrace Cluster version 127 or higher.

To activate Go support

  1. Go to Settings > Monitoring > Monitored technologies.
  2. Set the Go switch to the On position.

Monitor Go with Dynatrace

To illustrate how Go monitoring works, and to show you the available metrics that Dynatrace provides for Go applications, we’ll use an example influxdb database that receives requests via a load-test script. This example uses the out-of-the-box influxdb Ubuntu package binary. To begin monitoring, following installation of Dynatrace OneAgent and activation of Go support (see above), restart the influxdb service by running the command systemctl restart influxdb.

After the restart the influxd process front page

Following restart of the influxd process, the process view will automatically display Go specific metrics  (see example below). Note that the Go version that the executable was built with (1.7.4)  is listed under Properties.

Key metrics

Dynatrace OneAgent immediately begins tracking the following important Go metrics.

Suspension Go garbage collector percentage share compared to overall application CPU time.
Runtime heap The number of bytes used/committed to Go heap and Go garbage collector execution times.
Goroutine count The number of Go routines instantiated by the application and Go runtime infrastructure.

Click the Further details tab to drill down to the details of the running process.

Go managed memory

The Go managed memory tab breaks down memory metrics into various categories:

Heap Bytes of memory used/committed to Go runtime heap.
Stack Bytes of memory used/committed to dynamic Go stacks. Go stacks are used to execute Go routines and grow dynamically.
OffHeap Bytes of memory used/committed for Go runtime internal structures that are not allocated from heap memory. Data structures used in Go heap implementation are an example of OffHeap memory.
Overall The sum of Heap, OffHeap, and Stack memory.

Heap details

The Heap details tab digs deeper into the anatomy of Go heaps.

Heap allocated Go objects count Number of Go objects allocated on the heap.
Heap idle size Number of bytes currently not assigned to Go heap or stack. Idle memory may be returned to OS, or retained by Go runtime for later reassignment to heap or stack category.
Heep live size Number of bytes considered live by the Go garbage collector. This metric accumulates memory retained by the most recent garbage collector run and allocated since then.
Garbage collector invocation count Number of Go garbage collector runs.

Understand Goroutine scheduling

Go does a fantastic job of hiding Goroutine scheduling details from application developers. A basic understanding of the internal scheduling mechanics will help you read scheduling metrics and detect potential anomalies. As documentation of Go internals is sparse, following is a short primer to get you started.

The implementation of Goroutine scheduling deals with three central object types: M (Machine), P (Processor), and G (Goroutine). You’ll find many references to these object types by browsing through Go runtime source code. For expressiveness, we use the following alternative terms for M, P, and G:

Go runtime nomenclature Expressive name
M Worker thread
P Scheduling context
G Goroutine

Go executes Goroutines in context of worker threads acquired from a pool of native operating system threads. A Goroutine may be assigned to a different worker thread at any point in time (unless runtime.LockOSThread is used to enforce worker thread affinity).

Multiple Goroutines are typically assigned to a single worker thread. A scheduling context is responsible for the cooperative scheduling of these Goroutines. The Go compiler adds code to each Go function prologue which checks if the currently executing Goroutine consumed its 10 milliseconds execution time slice (the actual mechanism cleverly uses stack guards to enforce rescheduling). If the time slice is exceeded, the scheduling context sets up the next Goroutine to execute. This is why scheduling is cooperative: if a Goroutine doesn’t invoke Go functions, it won’t be rescheduled.

Each set of Goroutines is executed by a worker thread. Execution is controlled by a scheduling context. But what happens if a Goroutine writes a large chunk of data to disk or blocks waiting for an incoming connection? The Goroutine will be blocked in a system call and no other Goroutine will be scheduled (remember, reschedule happens only upon invocation of Go functions). Thus all Goroutines assigned to the same worker thread will also be blocked!

But no worries, Go deals elegantly with this situation. If a Goroutine executes a blocking system call, the scheduling context with the other Goroutines will be assigned on the fly to a different worker thread (either a parked or newly instantiated thread). This explains why the total number of worker threads is larger than the number of scheduling context objects. Once the blocking call returns, the Goroutine is again assigned to a scheduling context or, if the assignment fails, to the global Goroutine run queue. By the way, the very same principle applies to cgo (Go to C language) calls.

It’s notable that the number of scheduling contexts is the only user configurable setting in the Go scheduling algorithm (see the GOMAXPROCS environment variable and runtime.GOMAXPROCS function). You can’t control the number of worker threads.

Therefore, writing a large chunk of data to disk or waiting for incoming connections won’t block other Goroutines that are initially assigned to the same worker thread. These Goroutines will continue execution on their newly assigned worker threads.

With these fundamentals in mind, you’re now ready to dive into scheduling metrics.


The Scheduling tab provides unique insights into Goroutine scheduling.

System call count Number of system calls executed by Go runtime. The number does not include system calls performed in the context of cgo.
cgo call count Number of cgo calls.
Parked worker threads count Number of worker threads parked by Go runtime. A parked worker thread doesn’t consume CPU cycles until Go runtime unparks the thread.
“Out of work” worker threads count A worker thread is considered out of work when the associated scheduling context has no more Goroutines to execute. The worker thread attempts to steal Goroutines from another scheduling context and global run queue. If the stealing fails, the worker thread will park itself after some time.

The same mechanism applies to a high work load scenario. If an idle scheduling context exists, Go runtime will “unpark” a parked worker thread and associate the idle scheduling context. The unparked worker thread is now in the ‘out of work’ state and will start Goroutine stealing.

Worker thread count Number of operating system threads instantiated to execute Go routines. Go does not terminate worker threads; it keeps them in a parked state for future re-use.
Global Goroutine run queue size Number of Goroutines in the global run queue. Goroutines are placed in the global run queue if the worker thread used to execute a blocking system call can’t acquire a scheduling context. Scheduling contexts periodically acquire Goroutines from the global run queue.
Idle scheduling context count A scheduling context is considered idle if it has no more Goroutines to execute and Goroutine acquisition from the global run queue or other scheduling contexts have failed.

Future outlook

This initial beta release of Dynatrace Go monitoring provides some great metrics and Go version detection, but there’s much more to come! The next version of Dynatrace Go monitoring will feature full web-request tracing capabilities—similar to those currently offered for all other supported technologies. Our goal is to make Dynatrace the first and only full-stack monitoring solution in the world that supports end-to-end transaction monitoring of Go processes, with no required changes to your code. As Go adoption grows, this promises to be a game changer.


Many Dynatrace folks have made important contributions and valuable input to Dynatrace Go technology support. But one individual deserves special acknowledgment: Michael “Mr. Runtime” Obermueller. Our Go support would not have been possible without Michael’s valuable input and vast knowledge of Go internal mechanisms.

The post Introducing fully automated support for Go-based application monitoring appeared first on Dynatrace blog – monitoring redefined.

Read the original blog entry...

More Stories By Dynatrace Blog

Building a revolutionary approach to software performance monitoring takes an extraordinary team. With decades of combined experience and an impressive history of disruptive innovation, that’s exactly what we ruxit has.

Get to know ruxit, and get to know the future of data analytics.

Latest Stories
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, will introduce two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a mu...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
Mobile device usage has increased exponentially during the past several years, as consumers rely on handhelds for everything from news and weather to banking and purchases. What can we expect in the next few years? The way in which we interact with our devices will fundamentally change, as businesses leverage Artificial Intelligence. We already see this taking shape as businesses leverage AI for cost savings and customer responsiveness. This trend will continue, as AI is used for more sophistica...
SYS-CON Events announced today that SourceForge has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SourceForge is the largest, most trusted destination for Open Source Software development, collaboration, discovery and download on the web serving over 32 million viewers, 150 million downloads and over 460,000 active development projects each and every month.
"NetApp's vision is how we help organizations manage data - delivering the right data in the right place, in the right time, to the people who need it, and doing it agnostic to what the platform is," explained Josh Atwell, Developer Advocate for NetApp, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
What You Need to Know You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technolog...
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp em...
One of the biggest challenges with adopting a DevOps mentality is: new applications are easily adapted to cloud-native, microservice-based, or containerized architectures - they can be built for them - but old applications need complex refactoring. On the other hand, these new technologies can require relearning or adapting new, oftentimes more complex, methodologies and tools to be ready for production. In his general session at @DevOpsSummit at 20th Cloud Expo, Chris Brown, Solutions Marketi...
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.
Most of the time there is a lot of work involved to move to the cloud, and most of that isn't really related to AWS or Azure or Google Cloud. Before we talk about public cloud vendors and DevOps tools, there are usually several technical and non-technical challenges that are connected to it and that every company needs to solve to move to the cloud. In his session at 21st Cloud Expo, Stefano Bellasio, CEO and founder of Cloud Academy Inc., will discuss what the tools, disciplines, and cultural...
Why Federal cloud? What is in Federal Clouds and integrations? This session will identify the process and the FedRAMP initiative. But is it sufficient? What is the remedy for keeping abreast of cutting-edge technology? In his session at 21st Cloud Expo, Rasananda Behera will examine the proposed solutions: Private or public or hybrid cloud Responsible governing bodies How can we accomplish?
SYS-CON Events announced today that N3N will exhibit at SYS-CON's @ThingsExpo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. N3N’s solutions increase the effectiveness of operations and control centers, increase the value of IoT investments, and facilitate real-time operational decision making. N3N enables operations teams with a four dimensional digital “big board” that consolidates real-time live video feeds alongside IoT sensor data a...
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...
Real IoT production deployments running at scale are collecting sensor data from hundreds / thousands / millions of devices. The goal is to take business-critical actions on the real-time data and find insights from stored datasets. In his session at @ThingsExpo, John Walicki, Watson IoT Developer Advocate at IBM Cloud, will provide a fast-paced developer journey that follows the IoT sensor data from generation, to edge gateway, to edge analytics, to encryption, to the IBM Bluemix cloud, to Wa...