Welcome!

Blog Feed Post

AppD on AppD: Scaling Our Custom Dashboards Platform

This blog post is a developer’s perspective on how using our own AppDynamics software has helped us find and fix performance-related issues – and how other developers can do the same. 

One of the most challenging aspects of developing cloud-based platforms is scalability. As we innovate and build new features, it is essential for developers to ensure these new platform features are scalable and do not impact the performance of our applications, especially as any impact on performance is likely to impact all tenants on the cluster.

This is a challenge that we experienced firsthand. My team here at Appdynamics is responsible for building custom dashboards, one of the most powerful features of AppDynamics. It allows you to group relevant metrics into one central dashboard as well as build sophisticated dashboards with drill down capabilities.

As we added more capabilities and support for the different types of metrics, we realized that scaling our dashboards was becoming a challenge. Many of our customers were exploring different dashboard capabilities and features, which was truly stressing our endpoints. This is when our practice of using AppDynamics internally – “AppD on AppD” – helped give us insight and visibility into the issues we faced while developing new features on the platform.

Our AppD on AppD Approach

Recently, we made some changes on the backend of a REST endpoint that returns metrics for different types of widgets on custom dashboards. Our team’s internal process is to ensure every feature we implement is stress-tested under a real type of load profile, so we made sure to test this as well. Below is what we discovered.

On local dev environments, the feature worked fine and the data returned by the API took a few milliseconds. Next, we deployed our changes to our performance environment, which is similar to an actual production instance with large amounts of data. We immediately noticed that under stress, the UI which uses the endpoint was very slow and the average response time of the endpoint was considerably high. Luckily, since AppDynamics was monitoring the performance environment, it was easy for us to dig into the issue.

We configured a business transaction to monitor the REST endpoint with the slow response time and within a few minutes, we collected transaction snapshots which gave us valuable behind-the-scenes information of the REST API calls. There were two things that caught our attention:

Resolved Issue #1

A particular method which was being called repeatedly was taking more time to execute than before. The method itself took around 100-150 milliseconds, but if it was called 100 times in a single transaction, it would take around 15,000 milliseconds to execute, which is roughly 15 seconds.

The image below shows the total time it took for all the calls of this method in this single transaction.

Here is a code snippet of that method:

After a few minutes of looking at the full implementation of this method, we found that String.replace could be a potential problem here, which happens to be slower than StringUtils.replace. As a result, we made a minor change and modified the code to use StringUtil implementation. Here is some information on StringUtils.replace vs String.replace.

Resolved Issue #2

Another issue we noticed was that there were too many database calls for a single request.

We made a few optimizations here, including caching the data and bulking-up the queries. After applying these fixes, we measured the performance again and saw the average response time for the endpoint improve drastically.

Lastly, we created our own custom dashboard to measure performance of our specific endpoint, showing us different metrics like average response time, errors, and calls per minute along with thresholds and baselines. We also created a scheduled report that sends our team a dashboard snapshot everyday, allowing us to easily spot outliers and proactively address issues. We also could have created alerts, which notify you when certain conditions are met or exceeded based on your configuration. Scheduled reports and alerts make it easy to spot outliers and proactively address issues.

Without AppDynamics, it would have been difficult for a developer to quickly pinpoint these issues within a large codebase. But with AppDynamics, not only does it become easier to find issues in production, but developers can proactively ensure that features are robust and scalable before deploying them to production. This reduces the time spent on performance-related issues, and instead, gives developers more time to innovate and write code.

Learn more about our business-centric dashboards or get a demo from our sales team today.

The post AppD on AppD: Scaling Our Custom Dashboards Platform appeared first on Application Performance Monitoring Blog | AppDynamics.

Read the original blog entry...

More Stories By AppDynamics Blog

In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally.

DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.

Latest Stories
Every organization is facing their own Digital Transformation as they attempt to stay ahead of the competition, or worse, just keep up. Each new opportunity, whether embracing machine learning, IoT, or a cloud migration, seems to bring new development, deployment, and management models. The results are more diverse and federated computing models than any time in our history.
On-premise or off, you have powerful tools available to maximize the value of your infrastructure and you demand more visibility and operational control. Fortunately, data center management tools keep a vigil on memory contestation, power, thermal consumption, server health, and utilization, allowing better control no matter your cloud's shape. In this session, learn how Intel software tools enable real-time monitoring and precise management to lower operational costs and optimize infrastructure...
"Calligo is a cloud service provider with data privacy at the heart of what we do. We are a typical Infrastructure as a Service cloud provider but it's been designed around data privacy," explained Julian Box, CEO and co-founder of Calligo, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Isomorphic Software is the global leader in high-end, web-based business applications. We develop, market, and support the SmartClient & Smart GWT HTML5/Ajax platform, combining the productivity and performance of traditional desktop software with the simplicity and reach of the open web. With staff in 10 timezones, Isomorphic provides a global network of services related to our technology, with offerings ranging from turnkey application development to SLA-backed enterprise support. Leadin...
While a hybrid cloud can ease that transition, designing and deploy that hybrid cloud still offers challenges for organizations concerned about lack of available cloud skillsets within their organization. Managed service providers offer a unique opportunity to fill those gaps and get organizations of all sizes on a hybrid cloud that meets their comfort level, while delivering enhanced benefits for cost, efficiency, agility, mobility, and elasticity.
DevOps has long focused on reinventing the SDLC (e.g. with CI/CD, ARA, pipeline automation etc.), while reinvention of IT Ops has lagged. However, new approaches like Site Reliability Engineering, Observability, Containerization, Operations Analytics, and ML/AI are driving a resurgence of IT Ops. In this session our expert panel will focus on how these new ideas are [putting the Ops back in DevOps orbringing modern IT Ops to DevOps].
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understa...
Enterprises are striving to become digital businesses for differentiated innovation and customer-centricity. Traditionally, they focused on digitizing processes and paper workflow. To be a disruptor and compete against new players, they need to gain insight into business data and innovate at scale. Cloud and cognitive technologies can help them leverage hidden data in SAP/ERP systems to fuel their businesses to accelerate digital transformation success.
Most organizations are awash today in data and IT systems, yet they're still struggling mightily to use these invaluable assets to meet the rising demand for new digital solutions and customer experiences that drive innovation and growth. What's lacking are potent and effective ways to rapidly combine together on-premises IT and the numerous commercial clouds that the average organization has in place today into effective new business solutions.
Concerns about security, downtime and latency, budgets, and general unfamiliarity with cloud technologies continue to create hesitation for many organizations that truly need to be developing a cloud strategy. Hybrid cloud solutions are helping to elevate those concerns by enabling the combination or orchestration of two or more platforms, including on-premise infrastructure, private clouds and/or third-party, public cloud services. This gives organizations more comfort to begin their digital tr...
Keeping an application running at scale can be a daunting task. When do you need to add more capacity? Larger databases? Additional servers? These questions get harder as the complexity of your application grows. Microservice based architectures and cloud-based dynamic infrastructures are technologies that help you keep your application running with high availability, even during times of extreme scaling. But real cloud success, at scale, requires much more than a basic lift-and-shift migrati...
David Friend is the co-founder and CEO of Wasabi, the hot cloud storage company that delivers fast, low-cost, and reliable cloud storage. Prior to Wasabi, David co-founded Carbonite, one of the world's leading cloud backup companies. A successful tech entrepreneur for more than 30 years, David got his start at ARP Instruments, a manufacturer of synthesizers for rock bands, where he worked with leading musicians of the day like Stevie Wonder, Pete Townsend of The Who, and Led Zeppelin. David has ...
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understa...
Dion Hinchcliffe is an internationally recognized digital expert, bestselling book author, frequent keynote speaker, analyst, futurist, and transformation expert based in Washington, DC. He is currently Chief Strategy Officer at the industry-leading digital strategy and online community solutions firm, 7Summits.
Addteq is a leader in providing business solutions to Enterprise clients. Addteq has been in the business for more than 10 years. Through the use of DevOps automation, Addteq strives on creating innovative solutions to solve business processes. Clients depend on Addteq to modernize the software delivery process by providing Atlassian solutions, create custom add-ons, conduct training, offer hosting, perform DevOps services, and provide overall support services.