Welcome!

Blog Feed Post

Debugging CDNs: A Need for Change

It’s a Saturday afternoon: An email from a customer drove me to start troubleshooting an issue they were experiencing with Amazon S3. They were experiencing random high wait time, usually an indication of a problem with communication back to some sort of an origin system. Should’ve been a piece of cake, right?

Well, it wasn’t. Each CDN out there uses different debug headers and some don’t send anything back, and in this case, S3 was even more difficult to work with because the only thing available was a useless set of headers:

  • HTTP/1.1 200 OK
  • Cache-Control: max-age=2592000
  • Date: Sat, 18 Feb 2017 21:28:55 GMT
  • Content-Length: 66153
  • Content-Type: image/jpeg
  • Last-Modified: Wed, 20 May 2015 21:21:04 GMT
  • Accept-Ranges: bytes
  • ETag: “6a4e8ea14b77871a29798980aa09cece”
  • Server: AmazonS3
  • x-amz-id-2: ZhJWHmcfPWcsEjrFk9kXGrcVviEH3r3iGDSoYKc4mjjNe6W7T/x15ZVBnE+TT5aU1NZm29X0Xec=
  • x-amz-request-id: BB24FC949CCC93CC

My next thought was to just share the request ID with our customer so they could download some sort of log; however, AWS does not provide that information unless you are paying for support.

One can easily argue that S3 is not a CDN. On the S3 performance page, they ask you to use CloudFront for GET-intensive workload, though many people use S3 as an origin.

So, let’s take a look at S3 in comparison with other CDN providers when it comes to helping troubleshoot via the information available in the HTTP headers.

S3:

  • HTTP/1.1 200 OK
  • Cache-Control: max-age=2592000
  • Date: Sat, 18 Feb 2017 21:28:55 GMT
  • Content-Length: 66153
  • Content-Type: image/jpeg
  • Last-Modified: Wed, 20 May 2015 21:21:04 GMT
  • Accept-Ranges: bytes
  • ETag: “6a4e8ea14b77871a29798980aa09cece”
  • Server: AmazonS3
  • x-amz-id-2: ZhJWHmcfPWcsEjrFk9kXGrcVviEH3r3iGDSoYKc4mjjNe6W7T/x15ZVBnE+TT5aU1NZm29X0Xec=
  • x-amz-request-id: BB24FC949CCC93CC

 

Akamai:

If you pass this Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-get-request-id

This is what you get:

  • HTTP/1.1 200 OK
  • Server: Jetty(9.2.4.v20141028)
  • X-Request-Id: 238520
  • Content-Type: text/html; charset=UTF-8
  • Content-Language: en-US
  • Set-Edgescape-Cookie: true
  • Cache-Control: max-age=9821
  • Expires: Sun, 19 Feb 2017 04:40:52 GMT
  • Date: Sun, 19 Feb 2017 01:57:11 GMT
  • X-Cache: TCP_MEM_HIT from a184-51-206-135.deploy.akamaitechnologies.com (AkamaiGHost/8.2.2.1.1-19192351) (-)
  • Connection: keep-alive
  • X-Check-Cacheable: YES

Limelight:

  • HTTP/1.1 200 OK
  • Cache-Control: max-age=4
  • Connection: keep-alive
  • Date: Fri, 27 Jan 2017 18:47:22 GMT
  • Content-Length: 99450
  • Content-Type: audio/x-mpegurl
  • Expires: Fri, 27 Jan 2017 18:47:26 GMT
  • Last-Modified: Sat, 16 Mar 2013 21:41:29 GMT
  • Accept-Ranges: bytes
  • Server: Apache/2.4.12 (Unix)
  • X-Served-By: apache05.live01.hls.gen.med1.prod.mlbam.net
  • Access-Control-Allow-Origin: *
  • X-Cache: HIT from cds1351.lga.llnw.net c:0,c0:0,r:4,r0:0,s:5
  • X-Cache: HIT from cds1232.iad.llnw.net d:2,d0:0,c:10,c0:0,r:22,r0:0,s:22
  • X-Cache: MISS from sw.cds1294.iad.llnw.net

Fastly:

  • HTTP/1.1 200 OK
  • Content-Encoding: gzip
  • Content-Language: en
  • Content-Type: text/html; charset=utf-8
  • Link: ; rel=”canonical”,; rel=”shortlink”
  • Surrogate-Control: max-age=86400
  • Surrogate-Key: node node/2058
  • Via: 1.1 varnish
  • X-Content-Type-Options: nosniff
  • X-Frame-Options: SAMEORIGIN
  • Server: Artisanal bits
  • Last-Modified: Sat, 18 Feb 2017 20:48:21 GMT
  • Via: 1.1 varnish
  • Content-Length: 11329
  • Accept-Ranges: bytes
  • Date: Sun, 19 Feb 2017 01:58:24 GMT
  • Via: 1.1 varnish
  • Age: 18602
  • Connection: keep-alive
  • Fastly-Debug-Path: (D cache-lhr6351-LHR 1487469504) (F cache-lhr6320-LHR 1487450940) (D cache-sjc3128-SJC 1487450940) (F cache-sjc3120-SJC 1487450901)
  • Fastly-Debug-TTL: (H cache-lhr6351-LHR 67836.790 2592000.000 18602) (H cache-sjc3128-SJC – – 39)
  • Fastly-Debug-Digest: 09a954816c7b2f9614bfd284e92ed6cec21c76c5434c2deabab127826154417a
  • X-Served-By: cache-sjc3128-SJC, cache-lhr6351-LHR
  • X-Cache: HIT, HIT
  • X-Cache-Hits: 1, 5
  • X-Timer: S1487469504.425617,VS0,VE0
  • Vary: Cookie,fastly-ssl,Accept-Encoding
  • X-XSS-Protection: 1; mode=block
  • Cache-Control: max-age=0, private, must-revalidate
  • Strict-Transport-Security: max-age=31536000

 

Edgecast / Verizon Digital Media Services

  • HTTP/1.1 200 OK
  • Content-Encoding: gzip
  • Accept-Ranges: bytes
  • Alt-Svc: clear
  • Cache-Control: max-age=604800
  • Content-Type: text/css
  • Date: Sun, 19 Feb 2017 02:00:47 GMT
  • Expires: Sun, 26 Feb 2017 02:00:47 GMT
  • Last-Modified: Thu, 16 Feb 2017 06:52:22 GMT
  • Server: ECS (oxr/838B)
  • Strict-Transport-Security: max-age=31536000
  • Vary: Accept-Encoding
  • Via: 1.1 google
  • X-Cache: HIT
  • X-Served-By: us-central1-c
  • Content-Length: 479

 

CDNetworks:

  • HTTP/1.1 200 OK
  • Date: Sun, 19 Feb 2017 11:40:15 GMT
  • Server: PWS/8.2.0.5
  • Transfer-Encoding: chunked
  • X-Px: nc h0-s1027.p12-sjc ( origin)
  • Cache-Control: private
  • Content-Type: text/html; charset=UTF-8
  • Content-Encoding: gzip
  • Vary: Accept-Encoding
  • X-Zen-Fury: d22ebc323ffa4b916cbe2b078d0b36643cd7e07a
  • Link: <https://www.cdnetworks.com/wp-json/>; rel=”https://api.w.org/
  • Link: <https://www.cdnetworks.com/>; rel=shortlink
  • X-Pingback: https://www.cdnetworks.com/xmlrpc.php
  • X-Cache-Status: NOTCACHED
  • X-Hello-Human: If you want to work with us, please send e-mail to jobs at cdnetworks dot com or @CDNetworks on Twitter.
  • Connection: keep-alive

There is no consistency, and with more and more companies relying on multiple CDNs, it’s impossible to debug and troubleshoot.

In my opinion, there is a list of must-have HTTP headers:

  • Which Edge Server handled the request
  • Hit / Miss / and any other information (including hierarchical tiering/original shield information)
  • Time to Serve
  • Time to Fetch from origin
  • Transaction ID for further troubleshooting on the CDN side

Please, let’s all urge our CDN friends to adopt a common troubleshooting header structure (by the way, Limelight, could you not use the same x-header to display 3 different things?) and CDNetworks you are wasting bytes with that text to hire people!

Example:

  • X-Served-By: ABC
  • X-Status: HIT / MISS
  • X-Time: in milliseconds
  • X-Time-Origin: in millisecond
  • X-Transaction-ID

We thank all of the CDNs out there for making the internet faster and the user experience better, but the time has come to make time to troubleshoot faster and easier as well.

Mehdi – Catchpoint

 

The post Debugging CDNs: A Need for Change appeared first on Catchpoint's Blog.

Read the original blog entry...

More Stories By Mehdi Daoudi

Catchpoint radically transforms the way businesses manage, monitor, and test the performance of online applications. Truly understand and improve user experience with clear visibility into complex, distributed online systems.

Founded in 2008 by four DoubleClick / Google executives with a passion for speed, reliability and overall better online experiences, Catchpoint has now become the most innovative provider of web performance testing and monitoring solutions. We are a team with expertise in designing, building, operating, scaling and monitoring highly transactional Internet services used by thousands of companies and impacting the experience of millions of users. Catchpoint is funded by top-tier venture capital firm, Battery Ventures, which has invested in category leaders such as Akamai, Omniture (Adobe Systems), Optimizely, Tealium, BazaarVoice, Marketo and many more.

Latest Stories
China Unicom exhibit at the 19th International Cloud Expo, which took place at the Santa Clara Convention Center in Santa Clara, CA, in November 2016. China United Network Communications Group Co. Ltd ("China Unicom") was officially established in 2009 on the basis of the merger of former China Netcom and former China Unicom. China Unicom mainly operates a full range of telecommunications services including mobile broadband (GSM, WCDMA, LTE FDD, TD-LTE), fixed-line broadband, ICT, data communica...
Whether you like it or not, DevOps is on track for a remarkable alliance with security. The SEC didn’t approve the merger. And your boss hasn’t heard anything about it. Yet, this unruly triumvirate will soon dominate and deliver DevSecOps faster, cheaper, better, and on an unprecedented scale. In his session at DevOps Summit, Frank Bunger, VP of Customer Success at ScriptRock, discussed how this cathartic moment will propel the DevOps movement from such stuff as dreams are made on to a practic...
In their Live Hack” presentation at 17th Cloud Expo, Stephen Coty and Paul Fletcher, Chief Security Evangelists at Alert Logic, provided the audience with a chance to see a live demonstration of the common tools cyber attackers use to attack cloud and traditional IT systems. This “Live Hack” used open source attack tools that are free and available for download by anybody. Attendees learned where to find and how to operate these tools for the purpose of testing their own IT infrastructure. The...
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex softw...
SYS-CON Events announced today that Loom Systems will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2015, Loom Systems delivers an advanced AI solution to predict and prevent problems in the digital business. Loom stands alone in the industry as an AI analysis platform requiring no prior math knowledge from operators, leveraging the existing staff to succeed in the digital era. With offices in S...
SYS-CON Events announced today that Cloud Academy will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloud Academy is the industry’s most innovative, vendor-neutral cloud technology training platform. Cloud Academy provides continuous learning solutions for individuals and enterprise teams for Amazon Web Services, Microsoft Azure, Google Cloud Platform, and the most popular cloud computing technologies. Ge...
Historically, some banking activities such as trading have been relying heavily on analytics and cutting edge algorithmic tools. The coming of age of powerful data analytics solutions combined with the development of intelligent algorithms have created new opportunities for financial institutions. In his session at 20th Cloud Expo, Sebastien Meunier, Head of Digital for North America at Chappuis Halder & Co., will discuss how these tools can be leveraged to develop a lasting competitive advanta...
"My role is working with customers, helping them go through this digital transformation. I spend a lot of time talking to banks, big industries, manufacturers working through how they are integrating and transforming their IT platforms and moving them forward," explained William Morrish, General Manager Product Sales at Interoute, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
With billions of sensors deployed worldwide, the amount of machine-generated data will soon exceed what our networks can handle. But consumers and businesses will expect seamless experiences and real-time responsiveness. What does this mean for IoT devices and the infrastructure that supports them? More of the data will need to be handled at - or closer to - the devices themselves.
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
My team embarked on building a data lake for our sales and marketing data to better understand customer journeys. This required building a hybrid data pipeline to connect our cloud CRM with the new Hadoop Data Lake. One challenge is that IT was not in a position to provide support until we proved value and marketing did not have the experience, so we embarked on the journey ourselves within the product marketing team for our line of business within Progress. In his session at @BigDataExpo, Sum...
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, will posit that disruption is inevitable for c...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 20th International Cloud Expo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.