Welcome!

Blog Feed Post

AppD on AppD: Scaling Our Custom Dashboards Platform

This blog post is a developer’s perspective on how using our own AppDynamics software has helped us find and fix performance-related issues – and how other developers can do the same. 

One of the most challenging aspects of developing cloud-based platforms is scalability. As we innovate and build new features, it is essential for developers to ensure these new platform features are scalable and do not impact the performance of our applications, especially as any impact on performance is likely to impact all tenants on the cluster.

This is a challenge that we experienced firsthand. My team here at Appdynamics is responsible for building custom dashboards, one of the most powerful features of AppDynamics. It allows you to group relevant metrics into one central dashboard as well as build sophisticated dashboards with drill down capabilities.

As we added more capabilities and support for the different types of metrics, we realized that scaling our dashboards was becoming a challenge. Many of our customers were exploring different dashboard capabilities and features, which was truly stressing our endpoints. This is when our practice of using AppDynamics internally – “AppD on AppD” – helped give us insight and visibility into the issues we faced while developing new features on the platform.

Our AppD on AppD Approach

Recently, we made some changes on the backend of a REST endpoint that returns metrics for different types of widgets on custom dashboards. Our team’s internal process is to ensure every feature we implement is stress-tested under a real type of load profile, so we made sure to test this as well. Below is what we discovered.

On local dev environments, the feature worked fine and the data returned by the API took a few milliseconds. Next, we deployed our changes to our performance environment, which is similar to an actual production instance with large amounts of data. We immediately noticed that under stress, the UI which uses the endpoint was very slow and the average response time of the endpoint was considerably high. Luckily, since AppDynamics was monitoring the performance environment, it was easy for us to dig into the issue.

We configured a business transaction to monitor the REST endpoint with the slow response time and within a few minutes, we collected transaction snapshots which gave us valuable behind-the-scenes information of the REST API calls. There were two things that caught our attention:

Resolved Issue #1

A particular method which was being called repeatedly was taking more time to execute than before. The method itself took around 100-150 milliseconds, but if it was called 100 times in a single transaction, it would take around 15,000 milliseconds to execute, which is roughly 15 seconds.

The image below shows the total time it took for all the calls of this method in this single transaction.

Here is a code snippet of that method:

After a few minutes of looking at the full implementation of this method, we found that String.replace could be a potential problem here, which happens to be slower than StringUtils.replace. As a result, we made a minor change and modified the code to use StringUtil implementation. Here is some information on StringUtils.replace vs String.replace.

Resolved Issue #2

Another issue we noticed was that there were too many database calls for a single request.

We made a few optimizations here, including caching the data and bulking-up the queries. After applying these fixes, we measured the performance again and saw the average response time for the endpoint improve drastically.

Lastly, we created our own custom dashboard to measure performance of our specific endpoint, showing us different metrics like average response time, errors, and calls per minute along with thresholds and baselines. We also created a scheduled report that sends our team a dashboard snapshot everyday, allowing us to easily spot outliers and proactively address issues. We also could have created alerts, which notify you when certain conditions are met or exceeded based on your configuration. Scheduled reports and alerts make it easy to spot outliers and proactively address issues.

Without AppDynamics, it would have been difficult for a developer to quickly pinpoint these issues within a large codebase. But with AppDynamics, not only does it become easier to find issues in production, but developers can proactively ensure that features are robust and scalable before deploying them to production. This reduces the time spent on performance-related issues, and instead, gives developers more time to innovate and write code.

Learn more about our business-centric dashboards or get a demo from our sales team today.

The post AppD on AppD: Scaling Our Custom Dashboards Platform appeared first on Application Performance Monitoring Blog | AppDynamics.

Read the original blog entry...

More Stories By AppDynamics Blog

In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally.

DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.

Latest Stories
Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes. We are offering early bird savings...
"Calligo is a cloud service provider with data privacy at the heart of what we do. We are a typical Infrastructure as a Service cloud provider but it's been designed around data privacy," explained Julian Box, CEO and co-founder of Calligo, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
DXWorldEXPO LLC announced today that the upcoming DXWorldEXPO | DevOpsSUMMIT | CloudEXPO New York will feature 10 companies from Poland to participate at the "Poland Digital Transformation Pavilion" on November 12-13, 2018. Polish Digital Transformation companies which will exhibit at CloudEXPO | DevOpsSUMMIT | DXWorldEXPO include All in Mobile, dhosting, Cryptomage, Perfect Gym, Polcom, Apius Technologies, Aplisens, ELZAB SA, TELDAT, and Rebug.io.
Enterprises are universally struggling to understand where the new tools and methodologies of DevOps fit into their organizations, and are universally making the same mistakes. These mistakes are not unavoidable, and in fact, avoiding them gifts an organization with sustained competitive advantage, just like it did for Japanese Manufacturing Post WWII.
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments t...
There's no doubt that blockchain technology is a powerful tool for the enterprise, but bringing it mainstream has not been without challenges. As VP of Technology at 8base, Andrei is working to make developing a blockchain application accessible to anyone. With better tools, entrepreneurs and developers can work together to quickly and effectively launch applications that integrate smart contracts and blockchain technology. This will ultimately accelerate blockchain adoption on a global scale.
DXWorldEXPO LLC announced today that Nutanix has been named "Platinum Sponsor" of CloudEXPO | DevOpsSUMMIT | DXWorldEXPO New York, which will take place November 12-13, 2018 in New York City. Nutanix makes infrastructure invisible, elevating IT to focus on the applications and services that power their business. The Nutanix Enterprise Cloud Platform blends web-scale engineering and consumer-grade design to natively converge server, storage, virtualization and networking into a resilient, softwar...
Despite being the market leader, we recognized the need to transform and reinvent our business at Dynatrace, before someone else disrupted the market. Over the course of three years, we changed everything - our technology, our culture and our brand image. In this session we'll discuss how we navigated through our own innovator's dilemma, and share takeaways from our experience that you can apply to your own organization.
Founded in 2002 and headquartered in Chicago, Nexum® takes a comprehensive approach to security. Nexum approaches business with one simple statement: “Do what’s right for the customer and success will follow.” Nexum helps you mitigate risks, protect your data, increase business continuity and meet your unique business objectives by: Detecting and preventing network threats, intrusions and disruptions Equipping you with the information, tools, training and resources you need to effectively m...
Having been in the web hosting industry since 2002, dhosting has gained a great deal of experience while working on a wide range of projects. This experience has enabled the company to develop our amazing new product, which they are now excited to present! Among dHosting's greatest achievements, they can include the development of their own hosting panel, the building of their fully redundant server system, and the creation of dhHosting's unique product, Dynamic Edge.
The Transparent Cloud-computing Consortium (T-Cloud) is a neutral organization for researching new computing models and business opportunities in IoT era. In his session, Ikuo Nakagawa, Co-Founder and Board Member at Transparent Cloud Computing Consortium, will introduce the big change toward the "connected-economy" in the digital age. He'll introduce and describe some leading-edge business cases from his original points of view, and discuss models & strategies in the connected-economy. Nowad...
"DevOps is set to be one of the most profound disruptions to hit IT in decades," said Andi Mann. "It is a natural extension of cloud computing, and I have seen both firsthand and in independent research the fantastic results DevOps delivers. So I am excited to help the great team at @DevOpsSUMMIT and CloudEXPO tell the world how they can leverage this emerging disruptive trend."
For far too long technology teams have lived in siloes. Not only physical siloes, but cultural siloes pushed by competing objectives. This includes informational siloes where business users require one set of data and tech teams require different data. DevOps intends to bridge these gaps to make tech driven operations more aligned and efficient.
NanoVMs is the only production ready unikernel infrastructure solution on the market today. Unikernels prevent server intrusions by isolating applications to one virtual machine with no users, no shells and no way to run other programs on them. Unikernels run faster and are lighter than even docker containers.
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO Silicon Valley 2019 will cover all of these tools, with the most comprehensive program and with 222 rockstar speakers throughout our industry presenting 22 Keynotes and General Sessions, 250 Breakout Sessions along 10 Tracks, as well as our signature Power Panels. Our Expo Floor will bring together the leading global 200 companies throughout the world of Cloud Computing, DevOps, IoT, Smart Cities, FinTech, Digital Transformation, and all they entail. As ...