Welcome!

Blog Feed Post

Tips for Automated Smoke Tests Using Sauce Labs

It goes without saying the goal of adding functionality is to add a new feature without it disrupting everything else. It’s also nice when you fix a bug without it breaking the system. But, unfortunately most of us have encountered that awkward moment when a new check-in broke the functionality of our app. Don’t feel bad. It happens a lot.

Luckily we have regression tests to help us out, but in the business of dealing with browsers and end users, how do you deal with the complexity of over 500 combinations of operating system and browser varieties?

At AppDynamics, we use Sauce Labs and Selenium in our automated smoke tests to help do just that. Sauce Labs provides test automation across virtual machines of browsers and platforms. With Sauce Labs’ Platform Configurator, you first specify the API, device, OS, and browsers, then you’re a button click away from configuring the desired capabilities of your test.

Depending on your infrastructure and test environment, you can require the browser to have access to a server inside the firewall. Sauce Connect opens a channel between your local network and the remote browser in the testing environment in Sauce Labs, so the VM can access the test system’s server.

All the above is standard information, and you can find more details about using Sauce Labs from their Wiki. So let’s dig in and discuss some of the gotchas that we discovered using Sauce Labs, and how we deal with them.

Some challenges found when working with Sauce labs

1. Cache busting

Although Sauce Labs spins off pristine browsers for each test session, the browser can take a significant amount of time to start up. To amortize these costs, we reuse a single browser session for all the tests. Cached website data can skew the test results. If you want to get metrics such as page load time and DOM first byte ready, you need to make sure the resource under testing is not cached. When you instruct WebDriver to visit a URL, you need to consider the usage of some cache busting techniques, such as cache control headers and unique fragments. This way ensures a unique URL for each visit, and no confusing results will be generated.

This code is in Java with Selenium. We added UUID to the end of each URL request.

void loadURLInBrowser(String url, boolean reload) {

       UUID uuid = UUID.randomUUID();

       String hashString = "#" + uuid;

       browser.get(server.absoluteUrl(url  + hashString));

       server.beaconSaver.waitForExpectedEvents(beaconCount);

       if (reload) {

           browser.navigate().refresh();

       }

}

2. Unknown_ca errors when testing HTTPS

Browsers don’t send HTTPS request if they can’t find the right SSL Certificate. When the server uses a self-signed certificate (for the convenience of tests), the Sauce Labs browsers don’t trust this certificate. Instead, they modify the browser to trust their own certificate, which is installed in their VPN proxies. Newer versions of Safari and some other browsers are more secure and harder to modify, so they throw this error. Sauce Labs knows this issue, but hasn’t spent the time to figure out how to modify it. This creates problems, and you’ll see unknown_ca errors.

In our testing, we had this problem with:

  • Android 4.4 and 5.0

  • iOS 7.1 and later

  • Safari 8 and later

  • Edge

You have to avoid tests that depend on the Sauce Connect proxy for HTTPS on these browsers.

3. Clock synchronization issues between your local server and remote web browsers

Performance monitoring collects lots of timing metrics data, and we often need to validate a timestamp in our tests. However, the clock on the remote machine may not be synchronized with the clock on the local machine. The timestamps may differ significantly. When you design your tests, you need to keep the clock synchronization issue in mind. Likewise, network latency can be an issue. Requests and responses can arrive out of order, even when you try to add latency to responses. It’s best to skip tests that depend on the order of asynchronous requests.

4. Be sure to check a port’s availability     

If your test requires a distinct web server to go with each test, then you need to make sure the server is running and has the port open before you send the address to the browser. When starting tests in parallel in threads, the server could choose a port and report it to the browser before the port is opened, and it could fail because another server could open that port first.

5. Problematic tunnel ports: 6000 and 6666

If your pages load from localhost, then to be compatible with all browsers you need to use the ports tunneled by the Sauce Connect proxy in the VM, in accordance with the FAQ at Sauce Labs (https://wiki.saucelabs.com/display/DOCS/Sauce+Connect+Proxy+FAQS). However, we’ve empirically had issues with ports 6000 and 6666, so we removed those from our list. We also grouped the ports into pairs, so we could do HTTP/HTTPS tests.

public class SauceInfo {

   static final int[][] PORTS_TUNNELED_BY_SAUCE_CONNECT = {

           /* 6000 Safari doesn't want to connect to this, even when it's available (!?),*/

            /* 6666 Safari doesn't like ,*/

           {80, 443}, {2000, 2001}, {2020, 2109}, {2222, 2310}, {3000, 3001}, {3030, 3210}, {3333, 4000},

           {4001, 4040}, {4321, 4502}, {4503, 4567}, {5000, 5001}, {5050, 5432}, {6001, 6060}, {6543, 7000},

           {7070, 7774}, {7777, 8000}, {8001, 8003}, {8031, 8081}, {8765, 8777}, {8888, 9000}, {9001, 9080},

           {9090, 9876}, {9877, 9999}, {49221, 55001}

   };

}

Hope this article helps. Have fun playing with Sauce Labs in the smoke tests of your web apps.

The post Tips for Automated Smoke Tests Using Sauce Labs appeared first on Application Performance Monitoring Blog | AppDynamics.

Read the original blog entry...

More Stories By Jyoti Bansal

In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally.

DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.

Latest Stories
"I think DevOps is now a rambunctious teenager – it’s starting to get a mind of its own, wanting to get its own things but it still needs some adult supervision," explained Thomas Hooker, VP of marketing at CollabNet, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We are still a relatively small software house and we are focusing on certain industries like FinTech, med tech, energy and utilities. We help our customers with their digital transformation," noted Piotr Stawinski, Founder and CEO of EARP Integration, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"DX encompasses the continuing technology revolution, and is addressing society's most important issues throughout the entire $78 trillion 21st-century global economy," said Roger Strukhoff, Conference Chair. "DX World Expo has organized these issues along 10 tracks with more than 150 of the world's top speakers coming to Istanbul to help change the world."
"We've been engaging with a lot of customers including Panasonic, we've been involved with Cisco and now we're working with the U.S. government - the Department of Homeland Security," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We're here to tell the world about our cloud-scale infrastructure that we have at Juniper combined with the world-class security that we put into the cloud," explained Lisa Guess, VP of Systems Engineering at Juniper Networks, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"I will be talking about ChatOps and ChatOps as a way to solve some problems in the DevOps space," explained Himanshu Chhetri, CTO of Addteq, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We are focused on SAP running in the clouds, to make this super easy because we believe in the tremendous value of those powerful worlds - SAP and the cloud," explained Frank Stienhans, CTO of Ocean9, Inc., in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, provided a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services with...
The financial services market is one of the most data-driven industries in the world, yet it’s bogged down by legacy CPU technologies that simply can’t keep up with the task of querying and visualizing billions of records. In his session at 20th Cloud Expo, Karthik Lalithraj, a Principal Solutions Architect at Kinetica, discussed how the advent of advanced in-database analytics on the GPU makes it possible to run sophisticated data science workloads on the same database that is housing the rich...
SYS-CON Events announced today that Massive Networks will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Massive Networks mission is simple. To help your business operate seamlessly with fast, reliable, and secure internet and network solutions. Improve your customer's experience with outstanding connections to your cloud.
"We are an IT services solution provider and we sell software to support those solutions. Our focus and key areas are around security, enterprise monitoring, and continuous delivery optimization," noted John Balsavage, President of A&I Solutions, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution and join Akvelon expert and IoT industry leader, Sergey Grebnov, in his session at @ThingsExpo, for an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
"We want to show that our solution is far less expensive with a much better total cost of ownership so we announced several key features. One is called geo-distributed erasure coding, another is support for KVM and we introduced a new capability called Multi-Part," explained Tim Desai, Senior Product Marketing Manager at Hitachi Data Systems, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol,...
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...