Welcome!

Blog Feed Post

Tips for Automated Smoke Tests Using Sauce Labs

It goes without saying the goal of adding functionality is to add a new feature without it disrupting everything else. It’s also nice when you fix a bug without it breaking the system. But, unfortunately most of us have encountered that awkward moment when a new check-in broke the functionality of our app. Don’t feel bad. It happens a lot.

Luckily we have regression tests to help us out, but in the business of dealing with browsers and end users, how do you deal with the complexity of over 500 combinations of operating system and browser varieties?

At AppDynamics, we use Sauce Labs and Selenium in our automated smoke tests to help do just that. Sauce Labs provides test automation across virtual machines of browsers and platforms. With Sauce Labs’ Platform Configurator, you first specify the API, device, OS, and browsers, then you’re a button click away from configuring the desired capabilities of your test.

Depending on your infrastructure and test environment, you can require the browser to have access to a server inside the firewall. Sauce Connect opens a channel between your local network and the remote browser in the testing environment in Sauce Labs, so the VM can access the test system’s server.

All the above is standard information, and you can find more details about using Sauce Labs from their Wiki. So let’s dig in and discuss some of the gotchas that we discovered using Sauce Labs, and how we deal with them.

Some challenges found when working with Sauce labs

1. Cache busting

Although Sauce Labs spins off pristine browsers for each test session, the browser can take a significant amount of time to start up. To amortize these costs, we reuse a single browser session for all the tests. Cached website data can skew the test results. If you want to get metrics such as page load time and DOM first byte ready, you need to make sure the resource under testing is not cached. When you instruct WebDriver to visit a URL, you need to consider the usage of some cache busting techniques, such as cache control headers and unique fragments. This way ensures a unique URL for each visit, and no confusing results will be generated.

This code is in Java with Selenium. We added UUID to the end of each URL request.

void loadURLInBrowser(String url, boolean reload) {

       UUID uuid = UUID.randomUUID();

       String hashString = "#" + uuid;

       browser.get(server.absoluteUrl(url  + hashString));

       server.beaconSaver.waitForExpectedEvents(beaconCount);

       if (reload) {

           browser.navigate().refresh();

       }

}

2. Unknown_ca errors when testing HTTPS

Browsers don’t send HTTPS request if they can’t find the right SSL Certificate. When the server uses a self-signed certificate (for the convenience of tests), the Sauce Labs browsers don’t trust this certificate. Instead, they modify the browser to trust their own certificate, which is installed in their VPN proxies. Newer versions of Safari and some other browsers are more secure and harder to modify, so they throw this error. Sauce Labs knows this issue, but hasn’t spent the time to figure out how to modify it. This creates problems, and you’ll see unknown_ca errors.

In our testing, we had this problem with:

  • Android 4.4 and 5.0

  • iOS 7.1 and later

  • Safari 8 and later

  • Edge

You have to avoid tests that depend on the Sauce Connect proxy for HTTPS on these browsers.

3. Clock synchronization issues between your local server and remote web browsers

Performance monitoring collects lots of timing metrics data, and we often need to validate a timestamp in our tests. However, the clock on the remote machine may not be synchronized with the clock on the local machine. The timestamps may differ significantly. When you design your tests, you need to keep the clock synchronization issue in mind. Likewise, network latency can be an issue. Requests and responses can arrive out of order, even when you try to add latency to responses. It’s best to skip tests that depend on the order of asynchronous requests.

4. Be sure to check a port’s availability     

If your test requires a distinct web server to go with each test, then you need to make sure the server is running and has the port open before you send the address to the browser. When starting tests in parallel in threads, the server could choose a port and report it to the browser before the port is opened, and it could fail because another server could open that port first.

5. Problematic tunnel ports: 6000 and 6666

If your pages load from localhost, then to be compatible with all browsers you need to use the ports tunneled by the Sauce Connect proxy in the VM, in accordance with the FAQ at Sauce Labs (https://wiki.saucelabs.com/display/DOCS/Sauce+Connect+Proxy+FAQS). However, we’ve empirically had issues with ports 6000 and 6666, so we removed those from our list. We also grouped the ports into pairs, so we could do HTTP/HTTPS tests.

public class SauceInfo {

   static final int[][] PORTS_TUNNELED_BY_SAUCE_CONNECT = {

           /* 6000 Safari doesn't want to connect to this, even when it's available (!?),*/

            /* 6666 Safari doesn't like ,*/

           {80, 443}, {2000, 2001}, {2020, 2109}, {2222, 2310}, {3000, 3001}, {3030, 3210}, {3333, 4000},

           {4001, 4040}, {4321, 4502}, {4503, 4567}, {5000, 5001}, {5050, 5432}, {6001, 6060}, {6543, 7000},

           {7070, 7774}, {7777, 8000}, {8001, 8003}, {8031, 8081}, {8765, 8777}, {8888, 9000}, {9001, 9080},

           {9090, 9876}, {9877, 9999}, {49221, 55001}

   };

}

Hope this article helps. Have fun playing with Sauce Labs in the smoke tests of your web apps.

The post Tips for Automated Smoke Tests Using Sauce Labs appeared first on Application Performance Monitoring Blog | AppDynamics.

Read the original blog entry...

More Stories By Jyoti Bansal

In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally.

DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.

Latest Stories
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value S...
NHK, Japan Broadcasting, will feature the upcoming @ThingsExpo Silicon Valley in a special 'Internet of Things' and smart technology documentary that will be filmed on the expo floor between November 3 to 5, 2015, in Santa Clara. NHK is the sole public TV network in Japan equivalent to the BBC in the UK and the largest in Asia with many award-winning science and technology programs. Japanese TV is producing a documentary about IoT and Smart technology and will be covering @ThingsExpo Silicon Val...
Join IBM November 2 at 19th Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how to go beyond multi-speed it to bring agility to traditional enterprise applications. Technology innovation is the driving force behind modern business and enterprises must respond by increasing the speed and efficiency of software delivery. The challenge is that existing enterprise applications are expensive to develop and difficult to modernize. This often results in what Gartner calls ...
Translating agile methodology into real-world best practices within the modern software factory has driven widespread DevOps adoption, yet much work remains to expand workflows and tooling across the enterprise. As models evolve from pockets of experimentation into wholescale organizational reinvention, practitioners find themselves challenged to incorporate the culture and architecture necessary to support DevOps at scale. In his session at @DevOpsSummit at 20th Cloud Expo, Anand Akela, Senior...
@GonzalezCarmen has been ranked the Number One Influencer and @ThingsExpo has been named the Number One Brand in the “M2M 2016: Top 100 Influencers and Brands” by Analytic. Onalytica analyzed tweets over the last 6 months mentioning the keywords M2M OR “Machine to Machine.” They then identified the top 100 most influential brands and individuals leading the discussion on Twitter.
The 20th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held June 6-8, 2017, at the Javits Center in New York City, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Containers, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportunity. Submit your speaking proposal ...
The age of Digital Disruption is evolving into the next era – Digital Cohesion, an age in which applications securely self-assemble and deliver predictive services that continuously adapt to user behavior. Information from devices, sensors and applications around us will drive services seamlessly across mobile and fixed devices/infrastructure. This evolution is happening now in software defined services and secure networking. Four key drivers – Performance, Economics, Interoperability and Trust ...
NHK, Japan Broadcasting, will feature the upcoming @ThingsExpo Silicon Valley in a special 'Internet of Things' and smart technology documentary that will be filmed on the expo floor between November 3 to 5, 2015, in Santa Clara. NHK is the sole public TV network in Japan equivalent to the BBC in the UK and the largest in Asia with many award-winning science and technology programs. Japanese TV is producing a documentary about IoT and Smart technology and will be covering @ThingsExpo Silicon Val...
The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas. In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists discussed what things are the most important, which will have the most profound e...
SYS-CON Events announced today that Twistlock, the leading provider of cloud container security solutions, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Twistlock is the industry's first enterprise security suite for container security. Twistlock's technology addresses risks on the host and within the application of the container, enabling enterprises to consistently enforce security policies, monitor...
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will look at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deli...
Automation is enabling enterprises to design, deploy, and manage more complex, hybrid cloud environments. Yet the people who manage these environments must be trained in and understanding these environments better than ever before. A new era of analytics and cognitive computing is adding intelligence, but also more complexity, to these cloud environments. How smart is your cloud? How smart should it be? In this power panel at 20th Cloud Expo, moderated by Conference Chair Roger Strukhoff, pane...
With billions of sensors deployed worldwide, the amount of machine-generated data will soon exceed what our networks can handle. But consumers and businesses will expect seamless experiences and real-time responsiveness. What does this mean for IoT devices and the infrastructure that supports them? More of the data will need to be handled at - or closer to - the devices themselves.
Building a cross-cloud operational model can be a daunting task. Per-cloud silos are not the answer, but neither is a fully generic abstraction plane that strips out capabilities unique to a particular provider. In his session at 20th Cloud Expo, Chris Wolf, VP & Chief Technology Officer, Global Field & Industry at VMware, will discuss how successful organizations approach cloud operations and management, with insights into where operations should be centralized and when it’s best to decentraliz...
In recent years, containers have taken the world by storm. Companies of all sizes and industries have realized the massive benefits of containers, such as unprecedented mobility, higher hardware utilization, and increased flexibility and agility; however, many containers today are non-persistent. Containers without persistence miss out on many benefits, and in many cases simply pass the responsibility of persistence onto other infrastructure, adding additional complexity.