Welcome!

Blog Feed Post

Debugging CDNs: A Need for Change

It’s a Saturday afternoon: An email from a customer drove me to start troubleshooting an issue they were experiencing with Amazon S3. They were experiencing random high wait time, usually an indication of a problem with communication back to some sort of an origin system. Should’ve been a piece of cake, right?

Well, it wasn’t. Each CDN out there uses different debug headers and some don’t send anything back, and in this case, S3 was even more difficult to work with because the only thing available was a useless set of headers:

  • HTTP/1.1 200 OK
  • Cache-Control: max-age=2592000
  • Date: Sat, 18 Feb 2017 21:28:55 GMT
  • Content-Length: 66153
  • Content-Type: image/jpeg
  • Last-Modified: Wed, 20 May 2015 21:21:04 GMT
  • Accept-Ranges: bytes
  • ETag: “6a4e8ea14b77871a29798980aa09cece”
  • Server: AmazonS3
  • x-amz-id-2: ZhJWHmcfPWcsEjrFk9kXGrcVviEH3r3iGDSoYKc4mjjNe6W7T/x15ZVBnE+TT5aU1NZm29X0Xec=
  • x-amz-request-id: BB24FC949CCC93CC

My next thought was to just share the request ID with our customer so they could download some sort of log; however, AWS does not provide that information unless you are paying for support.

One can easily argue that S3 is not a CDN. On the S3 performance page, they ask you to use CloudFront for GET-intensive workload, though many people use S3 as an origin.

So, let’s take a look at S3 in comparison with other CDN providers when it comes to helping troubleshoot via the information available in the HTTP headers.

S3:

  • HTTP/1.1 200 OK
  • Cache-Control: max-age=2592000
  • Date: Sat, 18 Feb 2017 21:28:55 GMT
  • Content-Length: 66153
  • Content-Type: image/jpeg
  • Last-Modified: Wed, 20 May 2015 21:21:04 GMT
  • Accept-Ranges: bytes
  • ETag: “6a4e8ea14b77871a29798980aa09cece”
  • Server: AmazonS3
  • x-amz-id-2: ZhJWHmcfPWcsEjrFk9kXGrcVviEH3r3iGDSoYKc4mjjNe6W7T/x15ZVBnE+TT5aU1NZm29X0Xec=
  • x-amz-request-id: BB24FC949CCC93CC

 

Akamai:

If you pass this Pragma: akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-get-request-id

This is what you get:

  • HTTP/1.1 200 OK
  • Server: Jetty(9.2.4.v20141028)
  • X-Request-Id: 238520
  • Content-Type: text/html; charset=UTF-8
  • Content-Language: en-US
  • Set-Edgescape-Cookie: true
  • Cache-Control: max-age=9821
  • Expires: Sun, 19 Feb 2017 04:40:52 GMT
  • Date: Sun, 19 Feb 2017 01:57:11 GMT
  • X-Cache: TCP_MEM_HIT from a184-51-206-135.deploy.akamaitechnologies.com (AkamaiGHost/8.2.2.1.1-19192351) (-)
  • Connection: keep-alive
  • X-Check-Cacheable: YES

Limelight:

  • HTTP/1.1 200 OK
  • Cache-Control: max-age=4
  • Connection: keep-alive
  • Date: Fri, 27 Jan 2017 18:47:22 GMT
  • Content-Length: 99450
  • Content-Type: audio/x-mpegurl
  • Expires: Fri, 27 Jan 2017 18:47:26 GMT
  • Last-Modified: Sat, 16 Mar 2013 21:41:29 GMT
  • Accept-Ranges: bytes
  • Server: Apache/2.4.12 (Unix)
  • X-Served-By: apache05.live01.hls.gen.med1.prod.mlbam.net
  • Access-Control-Allow-Origin: *
  • X-Cache: HIT from cds1351.lga.llnw.net c:0,c0:0,r:4,r0:0,s:5
  • X-Cache: HIT from cds1232.iad.llnw.net d:2,d0:0,c:10,c0:0,r:22,r0:0,s:22
  • X-Cache: MISS from sw.cds1294.iad.llnw.net

Fastly:

  • HTTP/1.1 200 OK
  • Content-Encoding: gzip
  • Content-Language: en
  • Content-Type: text/html; charset=utf-8
  • Link: ; rel=”canonical”,; rel=”shortlink”
  • Surrogate-Control: max-age=86400
  • Surrogate-Key: node node/2058
  • Via: 1.1 varnish
  • X-Content-Type-Options: nosniff
  • X-Frame-Options: SAMEORIGIN
  • Server: Artisanal bits
  • Last-Modified: Sat, 18 Feb 2017 20:48:21 GMT
  • Via: 1.1 varnish
  • Content-Length: 11329
  • Accept-Ranges: bytes
  • Date: Sun, 19 Feb 2017 01:58:24 GMT
  • Via: 1.1 varnish
  • Age: 18602
  • Connection: keep-alive
  • Fastly-Debug-Path: (D cache-lhr6351-LHR 1487469504) (F cache-lhr6320-LHR 1487450940) (D cache-sjc3128-SJC 1487450940) (F cache-sjc3120-SJC 1487450901)
  • Fastly-Debug-TTL: (H cache-lhr6351-LHR 67836.790 2592000.000 18602) (H cache-sjc3128-SJC – – 39)
  • Fastly-Debug-Digest: 09a954816c7b2f9614bfd284e92ed6cec21c76c5434c2deabab127826154417a
  • X-Served-By: cache-sjc3128-SJC, cache-lhr6351-LHR
  • X-Cache: HIT, HIT
  • X-Cache-Hits: 1, 5
  • X-Timer: S1487469504.425617,VS0,VE0
  • Vary: Cookie,fastly-ssl,Accept-Encoding
  • X-XSS-Protection: 1; mode=block
  • Cache-Control: max-age=0, private, must-revalidate
  • Strict-Transport-Security: max-age=31536000

 

Edgecast / Verizon Digital Media Services

  • HTTP/1.1 200 OK
  • Content-Encoding: gzip
  • Accept-Ranges: bytes
  • Alt-Svc: clear
  • Cache-Control: max-age=604800
  • Content-Type: text/css
  • Date: Sun, 19 Feb 2017 02:00:47 GMT
  • Expires: Sun, 26 Feb 2017 02:00:47 GMT
  • Last-Modified: Thu, 16 Feb 2017 06:52:22 GMT
  • Server: ECS (oxr/838B)
  • Strict-Transport-Security: max-age=31536000
  • Vary: Accept-Encoding
  • Via: 1.1 google
  • X-Cache: HIT
  • X-Served-By: us-central1-c
  • Content-Length: 479

 

CDNetworks:

  • HTTP/1.1 200 OK
  • Date: Sun, 19 Feb 2017 11:40:15 GMT
  • Server: PWS/8.2.0.5
  • Transfer-Encoding: chunked
  • X-Px: nc h0-s1027.p12-sjc ( origin)
  • Cache-Control: private
  • Content-Type: text/html; charset=UTF-8
  • Content-Encoding: gzip
  • Vary: Accept-Encoding
  • X-Zen-Fury: d22ebc323ffa4b916cbe2b078d0b36643cd7e07a
  • Link: <https://www.cdnetworks.com/wp-json/>; rel=”https://api.w.org/
  • Link: <https://www.cdnetworks.com/>; rel=shortlink
  • X-Pingback: https://www.cdnetworks.com/xmlrpc.php
  • X-Cache-Status: NOTCACHED
  • X-Hello-Human: If you want to work with us, please send e-mail to jobs at cdnetworks dot com or @CDNetworks on Twitter.
  • Connection: keep-alive

There is no consistency, and with more and more companies relying on multiple CDNs, it’s impossible to debug and troubleshoot.

In my opinion, there is a list of must-have HTTP headers:

  • Which Edge Server handled the request
  • Hit / Miss / and any other information (including hierarchical tiering/original shield information)
  • Time to Serve
  • Time to Fetch from origin
  • Transaction ID for further troubleshooting on the CDN side

Please, let’s all urge our CDN friends to adopt a common troubleshooting header structure (by the way, Limelight, could you not use the same x-header to display 3 different things?) and CDNetworks you are wasting bytes with that text to hire people!

Example:

  • X-Served-By: ABC
  • X-Status: HIT / MISS
  • X-Time: in milliseconds
  • X-Time-Origin: in millisecond
  • X-Transaction-ID

We thank all of the CDNs out there for making the internet faster and the user experience better, but the time has come to make time to troubleshoot faster and easier as well.

Mehdi – Catchpoint

 

The post Debugging CDNs: A Need for Change appeared first on Catchpoint's Blog.

Read the original blog entry...

More Stories By Mehdi Daoudi

Catchpoint radically transforms the way businesses manage, monitor, and test the performance of online applications. Truly understand and improve user experience with clear visibility into complex, distributed online systems.

Founded in 2008 by four DoubleClick / Google executives with a passion for speed, reliability and overall better online experiences, Catchpoint has now become the most innovative provider of web performance testing and monitoring solutions. We are a team with expertise in designing, building, operating, scaling and monitoring highly transactional Internet services used by thousands of companies and impacting the experience of millions of users. Catchpoint is funded by top-tier venture capital firm, Battery Ventures, which has invested in category leaders such as Akamai, Omniture (Adobe Systems), Optimizely, Tealium, BazaarVoice, Marketo and many more.

Latest Stories
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
Blockchain is a shared, secure record of exchange that establishes trust, accountability and transparency across business networks. Supported by the Linux Foundation's open source, open-standards based Hyperledger Project, Blockchain has the potential to improve regulatory compliance, reduce cost as well as advance trade. Are you curious about how Blockchain is built for business? In her session at 21st Cloud Expo, René Bostic, Technical VP of the IBM Cloud Unit in North America, discussed the b...
The past few years have brought a sea change in the way applications are architected, developed, and consumed—increasing both the complexity of testing and the business impact of software failures. How can software testing professionals keep pace with modern application delivery, given the trends that impact both architectures (cloud, microservices, and APIs) and processes (DevOps, agile, and continuous delivery)? This is where continuous testing comes in. D
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone in...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
As you move to the cloud, your network should be efficient, secure, and easy to manage. An enterprise adopting a hybrid or public cloud needs systems and tools that provide: Agility: ability to deliver applications and services faster, even in complex hybrid environments Easier manageability: enable reliable connectivity with complete oversight as the data center network evolves Greater efficiency: eliminate wasted effort while reducing errors and optimize asset utilization Security: imple...
Mobile device usage has increased exponentially during the past several years, as consumers rely on handhelds for everything from news and weather to banking and purchases. What can we expect in the next few years? The way in which we interact with our devices will fundamentally change, as businesses leverage Artificial Intelligence. We already see this taking shape as businesses leverage AI for cost savings and customer responsiveness. This trend will continue, as AI is used for more sophistica...
No hype cycles or predictions of a gazillion things here. IoT is here. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, an Associate Partner of Analytics, IoT & Cybersecurity at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He also discussed the evaluation of communication standards and IoT messaging protocols, data...
Companies are harnessing data in ways we once associated with science fiction. Analysts have access to a plethora of visualization and reporting tools, but considering the vast amount of data businesses collect and limitations of CPUs, end users are forced to design their structures and systems with limitations. Until now. As the cloud toolkit to analyze data has evolved, GPUs have stepped in to massively parallel SQL, visualization and machine learning.
The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. Kubernetes was originally built by Google, leveraging years of experience with managing container workloads, and is now a Cloud Native Compute Foundation (CNCF) project. Kubernetes has been widely adopted by the community, supported on all major public and private cloud providers, and is gaining rapid adoption in enterprises. However, Kubernetes may seem intimidating and complex ...
In his session at 21st Cloud Expo, Michael Burley, a Senior Business Development Executive in IT Services at NetApp, described how NetApp designed a three-year program of work to migrate 25PB of a major telco's enterprise data to a new STaaS platform, and then secured a long-term contract to manage and operate the platform. This significant program blended the best of NetApp’s solutions and services capabilities to enable this telco’s successful adoption of private cloud storage and launching ...