Blog Feed Post

Hybrid Cloud Problem Patterns: Chasing DNS Lookup Times from AWS EC2

As a performance architect, I get called into various production performance issues. One of our recent production issues happened on Tomcat AppServer running on an AWS EC2 instance in a VPC. VPC is joined with an on-premise DNS server. This service calls another micro service. When service went live, we noticed a high response time from a downstream micro service, and the downstream service logs did not show any performance issue.

In this blog, I’ll walk through the steps taken by our tech arch Neeraj Verma to analyze this issue in our production environments, which tools were used, explaining background information DNS lookup, as well as how this problem was resolved. I hope you find this useful!

API High Response Time Analysis in Production Environment

Performance engineering is the science of discovering problem areas in applications under varying but realistic load conditions. It is not always easy to simulate real traffic and find all problems before going live. Therefore, it is advisable to determine out how to analyze performance problems, not only in test, but also in a real production environment. Having the right tools installed in production allows us to analyze issue and find root causes that are hard to simulate in testing.

The following diagram visualizes our service layout. Our eCommerce API’s call ShoppingHistory API using customerid. ShoppingHistory API calls DynamoDB and Customer- API-to-server requests from eCommerce.

Architectural Overview: Transactional Flow when the eCommerce Frontend calls the OrderAPI, and how it makes its way through different service layers deployed on AWS.

In order to monitor individual service health, we log entry and exit calls of each service invocation in a custom telemetry system. Each team uses Kibana/Grafana dashboards to measure health. Through the ShoppingHistory API dashboard, the team can see that time is being taken by Customer Service, even though the Customer Service dashboard did not show any issue at all. This is when the classical blame game would start. In our case we tasked the ShoppingHistory API team to find the actual root cause. And here is what we did.

Application Monitoring with Dynatrace AppMon

Our tool of choice was Dynatrace AppMon, which we already used to for live production performance monitoring of all our services in production. Let me walk you through the steps in Dynatrace AppMon on how we identified high response time and its root cause. In case you want to try it on your own I suggest you do the following:

  1. Get your own Dynatrace Personal License
  2. Watch the Dynatrace YouTube Tutorials
  3. Read up on Java Memory Management

Step #1: Basic Transaction

Once Dynatrace AppMon collects data you can decide whether to analyze it in the Dynatrace AppMon Diagnostics Client or go directly to the Dynatrace AppMon Web interface. With the recent improvements in Dynatrace AppMon 2017 May (v7) the Web Interface is even more convenient when analyzing PurePaths, which is why we go there. In the Web Interface we often start by looking at the Transaction Flow of our System. The Transaction Flow is dynamically generated by Dynatrace thanks to its capability to trace every single transaction, end-to-end, enabled through their PurePath technology.

Looking at the Transaction Flow, we could immediately see the most time (91%) was actually spent in ShoppingHistory JVM instead of Customer Service which we assumed until that point to be the problem as was indicated by our logging. Fortunately, Dynatrace AppMon told us otherwise!

The Dynatrace AppMon PurePath highlighted our Shopping History JVM as the response time hotspot.

Step 2: Drill Down into PurePath (show all nodes)

The detailed PurePath shows where most of the time is spent, down to the method itself. In our case we could spot that resolving the address of the backend microservice took about 2s. In the screenshot below you can see that when the frontend service tries to call the backend service it must first open a connection (HttpClient.connect method) which itself has to resolve the passed endpoint address. This method then calls the internal Java classes to do the actual DNS name resolution.

The PurePath Tree shows complete transaction flow, executed methods and how long they took to execute. Easy to spot the 2s execution time of the internal Java DNS name resolution method
The high level performance overview that you get for each PurePath also gives a good indication on which component our hotspot resides, clearly indicating the same problem — the amount of time spent making web request calls.


Based on the collected information from our production environment we tried finding a solution on the internet, and found an explanation for a similar issue on the IBM blog. This is where I found the answer to our issue:

The problem could be lookup issues between IPv6 versus IPv4. If the Domain Name System (DNS) server is not configured to handle IPv6 queries, the application may have to wait for the IPv6 query to timeout for IPv6 queries. By default, java.net.InetAddress resolutions requiring the network will be carried out over IPv6 if the operating system supports both IPv4 and IPv6. However, if the name service does not support IPv6, then a performance issue may be observed as the first IPv6 query has prolonged until its timeout before a successful IPv4 query can be made.

To reverse the default behavior and use IPv4 over IPv6, add the following Generic JVM argument:

  1. **-Djava.net.preferIPv4Stack=true**

We added -Djava.net.preferIPv4Stack=true  in JVM param and restarted JVM.

Now the transaction executed much faster, as shown in Dynatrace:

PurePath overview shows us a total execution time of 128ms. Most of the time now is spent in other areas – such as database calls – but no longer in resolving DNS addresses.
The PurePath Top Contributors tab makes this even clearer. HTTP calls now finish in milliseconds.
The API Breakdown also clearly shows that we solved this one problem. Now we can focus on the other hotspots if we want to improve performance further.

This story showed us how important it is to monitor all your applications and services in all the different environments on which they run. We will see a larger push towards hybrid cloud which means we have to find a way to detect these problems. Dynatrace natively supports all these technologies and, thanks to its analytics capabilities, makes it easy to find and fix them.

The post Hybrid Cloud Problem Patterns: Chasing DNS Lookup Times from AWS EC2 appeared first on Dynatrace blog – monitoring redefined.

Read the original blog entry...

More Stories By Dynatrace Blog

Building a revolutionary approach to software performance monitoring takes an extraordinary team. With decades of combined experience and an impressive history of disruptive innovation, that’s exactly what we ruxit has.

Get to know ruxit, and get to know the future of data analytics.

Latest Stories
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
The 21st International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will look at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deli...
While some vendors scramble to create and sell you a fancy solution for monitoring your spanking new Amazon Lambdas, hear how you can do it on the cheap using just built-in Java APIs yourself. By exploiting a little-known fact that Lambdas aren’t exactly single threaded, you can effectively identify hot spots in your serverless code. In his session at 20th Cloud Expo, David Martin, Principal Product Owner at CA Technologies, will give a live demonstration and code walkthrough, showing how to ov...
SYS-CON Events announced today that Systena America will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Systena Group has been in business for various software development and verification in Japan, US, ASEAN, and China by utilizing the knowledge we gained from all types of device development for various industries including smartphones (Android/iOS), wireless communication, security technology and IoT serv...
SYS-CON Events announced today that Loom Systems will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2015, Loom Systems delivers an advanced AI solution to predict and prevent problems in the digital business. Loom stands alone in the industry as an AI analysis platform requiring no prior math knowledge from operators, leveraging the existing staff to succeed in the digital era. With offices in S...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
SYS-CON Events announced today that EARP Integration will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. EARP Integration is a passionate software house. Since its inception in 2009 the company successfully delivers smart solutions for cities and factories that start their digital transformation. EARP provides bespoke solutions like, for example, advanced enterprise portals, business intelligence systems an...
We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA
SYS-CON Events announced today that Enzu will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive ad...
Existing Big Data solutions are mainly focused on the discovery and analysis of data. The solutions are scalable and highly available but tedious when swapping in and swapping out occurs in disarray and thrashing takes place. The resolution for thrashing through machine learning algorithms and support nomenclature is through simple techniques. Organizations that have been collecting large customer data are increasingly seeing the need to use the data for swapping in and out and thrashing occurs ...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that Interoute has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Interoute is the owner operator of Europe's largest network and a global cloud services platform, which encompasses over 70,000 km of lit fiber, 15 data centers, 17 virtual data centers and 33 colocation centers, with connections to 195 additional partner data centers. Our full-service Unifie...
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @CloudExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs oft...