Welcome!

Blog Feed Post

Debugging CDNs: A Need for Change

It’s a Saturday afternoon: An email from a customer drove me to start troubleshooting an issue they were experiencing with Amazon S3. They were experiencing random high wait time, usually an indication of a problem with communication back to some sort of an origin system. Should’ve been a piece of cake, right?

Well, it wasn’t. Each CDN out there uses different debug headers and some don’t send anything back, and in this case, S3 was even more difficult to work with because the only thing available was a useless set of headers:

  • HTTP/1.1 200 OK
  • Cache-Control: max-age=2592000
  • Date: Sat, 18 Feb 2017 21:28:55 GMT
  • Content-Length: 66153
  • Content-Type: image/jpeg
  • Last-Modified: Wed, 20 May 2015 21:21:04 GMT
  • Accept-Ranges: bytes
  • ETag: “6a4e8ea14b77871a29798980aa09cece”
  • Server: AmazonS3
  • x-amz-id-2: ZhJWHmcfPWcsEjrFk9kXGrcVviEH3r3iGDSoYKc4mjjNe6W7T/x15ZVBnE+TT5aU1NZm29X0Xec=
  • x-amz-request-id: BB24FC949CCC93CC

My next thought was to just share the request ID with our customer so they could download some sort of log; however, AWS does not provide that information unless you are paying for support.

One can easily argue that S3 is not a CDN. On the S3 performance page, they ask you to use CloudFront for GET-intensive workload, though many people use S3 as an origin.

So, let’s take a look at S3 in comparison with other CDN providers when it comes to helping troubleshoot via the information available in the HTTP headers.

S3:

  • HTTP/1.1 200 OK
  • Cache-Control: max-age=2592000
  • Date: Sat, 18 Feb 2017 21:28:55 GMT
  • Content-Length: 66153
  • Content-Type: image/jpeg
  • Last-Modified: Wed, 20 May 2015 21:21:04 GMT
  • Accept-Ranges: bytes
  • ETag: “6a4e8ea14b77871a29798980aa09cece”
  • Server: AmazonS3
  • x-amz-id-2: ZhJWHmcfPWcsEjrFk9kXGrcVviEH3r3iGDSoYKc4mjjNe6W7T/x15ZVBnE+TT5aU1NZm29X0Xec=
  • x-amz-request-id: BB24FC949CCC93CC

 

Akamai:

If you pass this Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-get-request-id

This is what you get:

  • HTTP/1.1 200 OK
  • Server: Jetty(9.2.4.v20141028)
  • X-Request-Id: 238520
  • Content-Type: text/html; charset=UTF-8
  • Content-Language: en-US
  • Set-Edgescape-Cookie: true
  • Cache-Control: max-age=9821
  • Expires: Sun, 19 Feb 2017 04:40:52 GMT
  • Date: Sun, 19 Feb 2017 01:57:11 GMT
  • X-Cache: TCP_MEM_HIT from a184-51-206-135.deploy.akamaitechnologies.com (AkamaiGHost/8.2.2.1.1-19192351) (-)
  • Connection: keep-alive
  • X-Check-Cacheable: YES

Limelight:

  • HTTP/1.1 200 OK
  • Cache-Control: max-age=4
  • Connection: keep-alive
  • Date: Fri, 27 Jan 2017 18:47:22 GMT
  • Content-Length: 99450
  • Content-Type: audio/x-mpegurl
  • Expires: Fri, 27 Jan 2017 18:47:26 GMT
  • Last-Modified: Sat, 16 Mar 2013 21:41:29 GMT
  • Accept-Ranges: bytes
  • Server: Apache/2.4.12 (Unix)
  • X-Served-By: apache05.live01.hls.gen.med1.prod.mlbam.net
  • Access-Control-Allow-Origin: *
  • X-Cache: HIT from cds1351.lga.llnw.net c:0,c0:0,r:4,r0:0,s:5
  • X-Cache: HIT from cds1232.iad.llnw.net d:2,d0:0,c:10,c0:0,r:22,r0:0,s:22
  • X-Cache: MISS from sw.cds1294.iad.llnw.net

Fastly:

  • HTTP/1.1 200 OK
  • Content-Encoding: gzip
  • Content-Language: en
  • Content-Type: text/html; charset=utf-8
  • Link: ; rel=”canonical”,; rel=”shortlink”
  • Surrogate-Control: max-age=86400
  • Surrogate-Key: node node/2058
  • Via: 1.1 varnish
  • X-Content-Type-Options: nosniff
  • X-Frame-Options: SAMEORIGIN
  • Server: Artisanal bits
  • Last-Modified: Sat, 18 Feb 2017 20:48:21 GMT
  • Via: 1.1 varnish
  • Content-Length: 11329
  • Accept-Ranges: bytes
  • Date: Sun, 19 Feb 2017 01:58:24 GMT
  • Via: 1.1 varnish
  • Age: 18602
  • Connection: keep-alive
  • Fastly-Debug-Path: (D cache-lhr6351-LHR 1487469504) (F cache-lhr6320-LHR 1487450940) (D cache-sjc3128-SJC 1487450940) (F cache-sjc3120-SJC 1487450901)
  • Fastly-Debug-TTL: (H cache-lhr6351-LHR 67836.790 2592000.000 18602) (H cache-sjc3128-SJC – – 39)
  • Fastly-Debug-Digest: 09a954816c7b2f9614bfd284e92ed6cec21c76c5434c2deabab127826154417a
  • X-Served-By: cache-sjc3128-SJC, cache-lhr6351-LHR
  • X-Cache: HIT, HIT
  • X-Cache-Hits: 1, 5
  • X-Timer: S1487469504.425617,VS0,VE0
  • Vary: Cookie,fastly-ssl,Accept-Encoding
  • X-XSS-Protection: 1; mode=block
  • Cache-Control: max-age=0, private, must-revalidate
  • Strict-Transport-Security: max-age=31536000

 

Edgecast / Verizon Digital Media Services

  • HTTP/1.1 200 OK
  • Content-Encoding: gzip
  • Accept-Ranges: bytes
  • Alt-Svc: clear
  • Cache-Control: max-age=604800
  • Content-Type: text/css
  • Date: Sun, 19 Feb 2017 02:00:47 GMT
  • Expires: Sun, 26 Feb 2017 02:00:47 GMT
  • Last-Modified: Thu, 16 Feb 2017 06:52:22 GMT
  • Server: ECS (oxr/838B)
  • Strict-Transport-Security: max-age=31536000
  • Vary: Accept-Encoding
  • Via: 1.1 google
  • X-Cache: HIT
  • X-Served-By: us-central1-c
  • Content-Length: 479

 

CDNetworks:

  • HTTP/1.1 200 OK
  • Date: Sun, 19 Feb 2017 11:40:15 GMT
  • Server: PWS/8.2.0.5
  • Transfer-Encoding: chunked
  • X-Px: nc h0-s1027.p12-sjc ( origin)
  • Cache-Control: private
  • Content-Type: text/html; charset=UTF-8
  • Content-Encoding: gzip
  • Vary: Accept-Encoding
  • X-Zen-Fury: d22ebc323ffa4b916cbe2b078d0b36643cd7e07a
  • Link: <https://www.cdnetworks.com/wp-json/>; rel=”https://api.w.org/
  • Link: <https://www.cdnetworks.com/>; rel=shortlink
  • X-Pingback: https://www.cdnetworks.com/xmlrpc.php
  • X-Cache-Status: NOTCACHED
  • X-Hello-Human: If you want to work with us, please send e-mail to jobs at cdnetworks dot com or @CDNetworks on Twitter.
  • Connection: keep-alive

There is no consistency, and with more and more companies relying on multiple CDNs, it’s impossible to debug and troubleshoot.

In my opinion, there is a list of must-have HTTP headers:

  • Which Edge Server handled the request
  • Hit / Miss / and any other information (including hierarchical tiering/original shield information)
  • Time to Serve
  • Time to Fetch from origin
  • Transaction ID for further troubleshooting on the CDN side

Please, let’s all urge our CDN friends to adopt a common troubleshooting header structure (by the way, Limelight, could you not use the same x-header to display 3 different things?) and CDNetworks you are wasting bytes with that text to hire people!

Example:

  • X-Served-By: ABC
  • X-Status: HIT / MISS
  • X-Time: in milliseconds
  • X-Time-Origin: in millisecond
  • X-Transaction-ID

We thank all of the CDNs out there for making the internet faster and the user experience better, but the time has come to make time to troubleshoot faster and easier as well.

Mehdi – Catchpoint

 

The post Debugging CDNs: A Need for Change appeared first on Catchpoint's Blog.

Read the original blog entry...

More Stories By Mehdi Daoudi

Catchpoint radically transforms the way businesses manage, monitor, and test the performance of online applications. Truly understand and improve user experience with clear visibility into complex, distributed online systems.

Founded in 2008 by four DoubleClick / Google executives with a passion for speed, reliability and overall better online experiences, Catchpoint has now become the most innovative provider of web performance testing and monitoring solutions. We are a team with expertise in designing, building, operating, scaling and monitoring highly transactional Internet services used by thousands of companies and impacting the experience of millions of users. Catchpoint is funded by top-tier venture capital firm, Battery Ventures, which has invested in category leaders such as Akamai, Omniture (Adobe Systems), Optimizely, Tealium, BazaarVoice, Marketo and many more.

Latest Stories
We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
SYS-CON Events announced today that SourceForge has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SourceForge is the largest, most trusted destination for Open Source Software development, collaboration, discovery and download on the web serving over 32 million viewers, 150 million downloads and over 460,000 active development projects each and every month.
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists looked at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deliver...
"We do one of the best file systems in the world. We learned how to deal with Big Data many years ago and we implemented this knowledge into our software," explained Jakub Ratajczak, Business Development Manager at MooseFS, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"Tintri focuses on the Ops side of the DevOps, which basically is pushing more and more of the accessibility of the infrastructure to the developers and trying to get behind the scenes," explained Dhiraj Sehgal of Tintri in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Cloud applications are seeing a deluge of requests to support the exploding advanced analytics market. “Open analytics” is the emerging strategy to deliver that data through an open data access layer, in the cloud, to be directly consumed by external analytics tools and popular programming languages. An increasing number of data engineers and data scientists use a variety of platforms and advanced analytics languages such as SAS, R, Python and Java, as well as frameworks such as Hadoop and Spark...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...
Both SaaS vendors and SaaS buyers are going “all-in” to hyperscale IaaS platforms such as AWS, which is disrupting the SaaS value proposition. Why should the enterprise SaaS consumer pay for the SaaS service if their data is resident in adjacent AWS S3 buckets? If both SaaS sellers and buyers are using the same cloud tools, automation and pay-per-transaction model offered by IaaS platforms, then why not host the “shrink-wrapped” software in the customers’ cloud? Further, serverless computing, cl...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
SYS-CON Events announced today that Enzu will exhibit at SYS-CON's 21st Int\ernational Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive advantage. By offering a suite of proven hosting and management services, Enzu wants companies to focus on the core of their ...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), provided an overview of various initiatives to certify the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldwide re...