Welcome!

Blog Feed Post

Using the Right Mean for Meaningful Performance Analysis

Performance analytics is a field which deals with huge discrete data sets that need to be grouped, organized, and aggregated to gain an understanding of the data. Synthetic and real user monitoring are the two most popular techniques to evaluate the performance of websites; both these techniques use historical data sets to evaluate performance.

In web performance analytics, it is preferred to use statistical values that describe a central tendency ( the odd numbermeasure of central location) for the discrete data set under observation. The statistical metric can be used to evaluate and analyze the data. These data sets have innumerable data points that need to be aggregated using different statistical approaches.

With the number of statistical metrics available, the big question is how do you determine the right statistical metric for a given data set. Mean, Median, and Geometric Mean are all valid measures of central tendency, but under different conditions, some measures of central tendency are more appropriate to use than others.

This article discusses different statistical approaches used in the world of web performance evaluation and the methods preferred in different contexts of performance analysis using real-world performance data.

Common Statistical Metrics

  • Arithmetic Mean (Average)

The average is used to describe a single central value in a large set of discrete data. The mathematical formula to calculate the average isThe average is equal to the sum of all data points divided by the number of items, where ‘n’ represents the number of data samples.

  • Median

Median is the middle score for a set of data that has been arranged in the order of magnitude. Let us consider a set of data point as [12, 31, 44, 47, 22, 18, 60, 75, 80]. To get the median of the data set the data points need to be sorted in ascending order.

12, 18, 22, 31, 44, 47, 60, 75, 80

The median for the above data set is ’44’ as the middle item is (n+1)/2 if odd number of items. The median would be n/2 if there is even number of items in the series.

  • Geometric Mean

Geometric mean is the nth positive root of the product of n positive given values. The mathematical formula to calculate the geometric mean for X containing n discrete set of data points is

  • Standard Deviation

Standard deviation is used for measuring the extent of variation of the data samples around the center. The mathematical formulae to calculate the standard deviation for a set of data samples is

Where ‘a’ denotes the average of ‘n’ data samples of value ‘x’.

Determining the Right Statistical Approach

The two graphs below illustrate the different data distributions we come across in web performance monitoring. Using the formulae explained above, we have derived the average, median and the geometric mean of the webpage load time for website A and B.

Webpage load time Website A

http://blog.catchpoint.com/wp-content/uploads/2017/05/stat4-300x130.png 300w, http://blog.catchpoint.com/wp-content/uploads/2017/05/stat4-768x332.png 768w" sizes="(max-width: 993px) 100vw, 993px" />

 Webpage load time Website B

http://blog.catchpoint.com/wp-content/uploads/2017/05/stat5-300x123.png 300w, http://blog.catchpoint.com/wp-content/uploads/2017/05/stat5-768x314.png 768w" sizes="(max-width: 994px) 100vw, 994px" />

Let us discuss a few use cases to understand how different statistical metrics are applicable in different scenarios.

USE CASE 1

G1 – Scatter plot showing webpage load time data set

http://blog.catchpoint.com/wp-content/uploads/2017/05/stat6-300x227.png 300w" sizes="(max-width: 500px) 100vw, 500px" />

G2 – Histogram shown the distribution of data

http://blog.catchpoint.com/wp-content/uploads/2017/05/stat7-300x223.png 300w" sizes="(max-width: 492px) 100vw, 492px" />

The graphs G1 and G2 plots data for webpage load time. The uneven distribution of the data points in the scatterplot and histogram helps us understand how inconsistent the load time is.

We can see a higher number of data points in the trailing end of the Gaussian distribution in the histogram (G2); this means that most of the data points are of higher value.

What would be a good statistical metric in such cases? Before answering this, lets us take an example. Consider the following data set

Data Set = [4,4.3,5,6.5,6.8,7,7.2,20,30]

If we use median it gives a value of 6.8. But most of the data points tend towards a higher range with 30 being the highest. So, taking the median value in cases with higher outliers is not an accurate estimate of the page load time. Median should be used for data sets with fewer outliers and values that are concentrated towards the center of the Gaussian distribution.

Now let us take the average for this same data set. This gives us a value of 27.4 which is slightly more skewed towards the outlier values. Once again, the average is not an accurate measure for web page load time.

Since median and average don’t apply to this set of data, let us consider the geometric mean. We get a value of 7.8 using geometric mean; this value is closer to the central value and is not skewed to the higher or lower values in the data set.

In this use case, we have determined the geometric mean as the most accurate statistical method to analyze the data.

USE CASE 2

G3 – Scatter plot showing webpage load time data set

http://blog.catchpoint.com/wp-content/uploads/2017/05/stat8-300x194.png 300w" sizes="(max-width: 462px) 100vw, 462px" />

G4 – Histogram shown the distribution of data

http://blog.catchpoint.com/wp-content/uploads/2017/05/stat9-300x230.png 300w" sizes="(max-width: 483px) 100vw, 483px" />

In the graphs above (G3 and G4), most of the data points are close to each other with a higher population in the center of Gaussian surface. The difference between each of the data points are much less than the distribution considered in the previous scenario. This indicates a consistent page load time across different test runs.

Using average or median to evaluate the central tendency would be more accurate in this case as there are not many outliers so the average wouldn’t be skewed towards the outlier values.

USE CASE 3

  Website A

http://blog.catchpoint.com/wp-content/uploads/2017/05/stat10-300x123.png 300w, http://blog.catchpoint.com/wp-content/uploads/2017/05/stat10-768x314.png 768w" sizes="(max-width: 832px) 100vw, 832px" />

Website B

http://blog.catchpoint.com/wp-content/uploads/2017/05/stat11-300x123.png 300w, http://blog.catchpoint.com/wp-content/uploads/2017/05/stat11-768x316.png 768w" sizes="(max-width: 827px) 100vw, 827px" />

 

The above data distribution shows the webpage load time for two different websites. In performance analysis, we need to evaluate the consistency of a webpage. And if there is high volatility in the page performance then we should be able to measure the difference between the central value versus the outliers.

In this case, the standard deviation values are 9.1 and 1.7 seconds for website A and B respectively while the median for website A and B are 26.6 and 18.1 seconds. Based on the standard deviation values, we see there are data points for website A at 36 secs (median + SD) and website B at 20 secs (median + SD). This means that website A had high number of data points concentrated at 36 secs or more and website B had high number data points concentrated at 20 secs or more.

To know what percent of data had higher value when compared to the standard deviation we can use the cumulative distribution graph.

Website A                                                                     
http://blog.catchpoint.com/wp-content/uploads/2017/05/stat12-300x127.png 300w, http://blog.catchpoint.com/wp-content/uploads/2017/05/stat12-768x324.png 768w" sizes="(max-width: 850px) 100vw, 850px" />
 Website B

http://blog.catchpoint.com/wp-content/uploads/2017/05/stat13-300x127.png 300w, http://blog.catchpoint.com/wp-content/uploads/2017/05/stat13-768x325.png 768w" sizes="(max-width: 827px) 100vw, 827px" />

From the cumulative distribution graph shown above we can see that website A had almost 20% of data points higher than the standard deviation values whereas website B had 10% of data more than standard deviation value.

Standard deviation can be used for evaluating how far and consistent the data points are with respect to the central value of data distribution in performance analysis.

 

Median and average are applicable when the data points are concentrated towards the center of the Gaussian distribution. On the other hand, if there are more data points distributed towards the tail of the Gaussian distribution and there is a high difference between each data point then geometric mean would be a better choice. Standard deviation should be used to understand the variance of the data points from the median value and to gauge the consistency of the sites performance.

 

The post Using the Right Mean for Meaningful Performance Analysis appeared first on Catchpoint's Blog - Web Performance Monitoring.

Read the original blog entry...

More Stories By Mehdi Daoudi

Catchpoint radically transforms the way businesses manage, monitor, and test the performance of online applications. Truly understand and improve user experience with clear visibility into complex, distributed online systems.

Founded in 2008 by four DoubleClick / Google executives with a passion for speed, reliability and overall better online experiences, Catchpoint has now become the most innovative provider of web performance testing and monitoring solutions. We are a team with expertise in designing, building, operating, scaling and monitoring highly transactional Internet services used by thousands of companies and impacting the experience of millions of users. Catchpoint is funded by top-tier venture capital firm, Battery Ventures, which has invested in category leaders such as Akamai, Omniture (Adobe Systems), Optimizely, Tealium, BazaarVoice, Marketo and many more.

Latest Stories
IBM helps FinTechs and financial services companies build and monetize cognitive-enabled financial services apps quickly and at scale. Hosted on IBM Bluemix, IBM’s platform builds in customer insights, regulatory compliance analytics and security to help reduce development time and testing. In his session at 20th Cloud Expo, Tom Eck, Industry Platforms CTO at IBM Cloud, will discuss how these tools simplify the time-consuming tasks of selection, mapping and data integration, allowing developers ...
In order to meet the rapidly changing demands of today’s customers, companies are continually forced to redefine their business strategies in order to meet these needs, stay relevant and continue to see profitable growth. IoT deployment and development is integral in this transformation, and today businesses are increasingly seeing the value of investing their resources into IoT deployments. These technologies are able increase ROI through projects such as connecting supply chains or enabling sm...
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs ofte...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that DivvyCloud will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. DivvyCloud software enables organizations to achieve their cloud computing goals by simplifying and automating security, compliance and cost optimization of public and private cloud infrastructure. Using DivvyCloud, customers can leverage programmatic Bots to identify and remediate common cloud problems in rea...
Interested in leveling up on your Cloud Foundry skills? Join IBM for Cloud Foundry Days on June 7 at Cloud Expo New York at the Javits Center in New York City. Cloud Foundry Days is a free half day educational conference and networking event. Come find out why Cloud Foundry is the industry's fastest-growing and most adopted cloud application platform.
For financial firms, the cloud is going to increasingly become a crucial part of dealing with customers over the next five years and beyond, particularly with the growing use and acceptance of virtual currencies. There are new data storage paradigms on the horizon that will deliver secure solutions for storing and moving sensitive financial data around the world without touching terrestrial networks. In his session at 20th Cloud Expo, Cliff Beek, President of Cloud Constellation Corporation, w...
While some vendors scramble to create and sell you a fancy solution for monitoring your spanking new Amazon Lambdas, hear how you can do it on the cheap using just built-in Java APIs yourself. By exploiting a little-known fact that Lambdas aren’t exactly single threaded, you can effectively identify hot spots in your serverless code. In his session at 20th Cloud Expo, David Martin, Principal Product Owner at CA Technologies, will give a live demonstration and code walkthrough, showing how to ov...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
SYS-CON Events announced today that Outscale, a global pure play Infrastructure as a Service provider and strategic partner of Dassault Systèmes, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2010, Outscale simplifies infrastructure complexities and boosts the business agility of its customers. Outscale delivers a secure, reliable and industrial strength solution for its customers, which in...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a ...
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet th...
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory?
SYS-CON Events announced today that A&I Solutions has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 1999, A&I Solutions is a leading information technology (IT) software and services provider focusing on best-in-class enterprise solutions. By partnering with industry leaders in technology, A&I assures customers high performance levels across all IT environments including: mai...
Every successful software product evolves from an idea to an enterprise system. Notably, the same way is passed by the product owner's company. In his session at 20th Cloud Expo, Oleg Lola, CEO of MobiDev, will provide a generalized overview of the evolution of a software product, the product owner, the needs that arise at various stages of this process, and the value brought by a software development partner to the product owner as a response to these needs.