Welcome!

Blog Feed Post

Continuous Testing at the Speed of DevOps

"Even if you've got a Maserati, you need a good driver who knows how to drive it. Speed is important, but safety and accuracy are also key" – Bob Aiello

Parasoft recently hosted a "Continuous Testing in the DevOps World" webinar featuring Bob Aiello, Technical Editor for CM Crossroads and Wayne Ariola, Parasoft Chief Strategy Officer and co-author of Continuous Testing. Since the webinar generated such overwhelming attendance and response, we thought we'd highlight some of the key points here.
 

What is DevOps?

DevOps is a set of principles and practices that helps you communicate and collaborate more effectively. Why? Because developers know a lot of very important information. So do QA testers… and so does Operations. If your organization is going to be successful, you need to be able to harness all of that knowledge and experience. A key part of DevOps is creating the "automated deployment pipeline" –the ability to deploy changes (bug fixes, new features, infrastructure changes, etc.) as often as needed, with absolute reliability. The ability to get it right each and every time is really key. Companies that master this gain a very important strategic advantage. Those that don't incur a huge amount of risk.

What is Continuous Testing?

Per the Continuous Testing Wikipedia page:

"Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. For Continuous Testing, the scope of testing extends from validating bottom-up requirements or user stories to assessing the system requirements associated with overarching business goals."

Are we done testing? Wrong question!

When assessing the risk of a release candidate, most software teams ask the question:  “Are we done testing?”  Fundamentally, this is the wrong question. 

Especially with DevOps and Continuous Delivery, releasing with both speed and confidence requires having immediate feedback on the business risks associated with a software release candidate. Given the rising cost and impact of software failures, you can't afford to unleash a release that could disrupt the existing user experience or introduce new features that expose the organization to new security, reliability, or compliance risks. To prevent this, the organization needs to extend from validating bottom-up requirements to assessing the system requirements associated with overarching business goals.

Instead of “Are we done testing,” software development teams should be asking: “Does the release candidate have an acceptable level of business risk?” If we can answer this question, we can determine if the application is truly ready to progress through the delivery pipeline at any given time.

What will the next generation of testing look like?

The overarching goal will be to close the gap between business expectations and dev/test activities. First off, we'll have to bring operations (runtime) data associated with the actual user experience into our testing environment. We'll also use simulation to enable testing to begin early and continue to run at the necessary frequency. We won't stop testing requirements or performing bottom-up verification of changes. This still needs to occur, but it needs to be designed into a context so that it's integrated into a continuous regression suite that's validated vs. the user experience. Understanding how new functionality impacts the broader system is key for accurately assessing the total risk of the release candidate.

All tests need to be placed into a regression suite that measures business objectives, then "process intelligence" can be used to better understand the impact of change. This will guide us to two things. First, better exploratory testing, where corner cases and more investigative testing can happen on more variable outcomes. Second, automated acceptance testing. This is designed to ensure that there's no negative impact to today's user experience.
 

What are key factors in moving from automated testing to Continuous Testing?

Automated testing involves automated, CI-driven execution of whatever set of tests the team has accumulated. However, if one of these tests fails, what does that really mean: does it indicate a critical business risk, or just a violation of some naming standard that nobody is really committed to following anyway? And what happens when it fails? Is there a clear workflow for prioritizing defects vs. business risks and addressing the most critical ones first? And for each defect that warrants fixing, is there a process for exposing all similar defects that might already have been introduced, as well as preventing this same problem from recurring in the future?  This is where the difference between automated and continuous becomes evident.

To evolve from automated to continuous, you need the following:

  1. Clearly defined business expectations, with business risks identified per application, team, and release.
  2. Defects automatically prioritized versus the business drivers and knowing how to mitigate those risks before the release candidate goes live.
  3. Testing in complete test environments continuously using simulation—this is critical for protecting the current user experience from the impact of change.
  4. Feedback loop for defect prevention—looking for patterns that emerge and using this as an opportunity to design and implement defect prevention practices that prevent similar defects from being introduced.
      

How does Service Virtualization fit into DevOps and Continuous Testing?

After organizations start accelerating their software delivery pipeline, they often reach the point where they need to test, but can't exercise the AUT because a complete test environment is not yet ready. A lot of teams use simulation technologies such as service virtualization to get around these roadblocks. For example, say your mainframe is only available for 2 hours on Saturday night. You can record your AUT's interactions with it, then capture this as a virtual asset. You can do the same for a database, 3rd-party application, SAP, etc.

Simulation technologies are not perfect. But in order for us to start to exercise the "big blocks" and get those risks out of the way, we need to begin exploratory testing . . . and we can't do this without simulation. To truly protect the end user experience, we need to aggressively test and defend the end user's experience across key end-to-end transactions. With today's systems, those transactions pass through a high number of different components, so it's very difficult to accommodate that in a single staged test environment—cloud or not. Simulation helps us get around this.  For the most realistic simulated environment, we need to really understand how components are working in an operational environment and transfer this to the simulation.

Watch the "Continuous Testing in the DevOps World" Webinar On Demand

Want to learn more? You can watch the complete 60-minute webinar at your convenience

 

Read the original blog entry...

More Stories By Wayne Ariola

Wayne Ariola is Vice President of Strategy and Corporate Development at Parasoft, a leading provider of integrated software development management, quality lifecycle management, and dev/test environment management solutions. He leverages customer input and fosters partnerships with industry leaders to ensure that Parasoft solutions continuously evolve to support the ever-changing complexities of real-world business processes and systems. Ariola has more than 15 years of strategic consulting experience within the technology and software development industries. He holds a BA from the University of California at Santa Barbara and an MBA from Indiana University.

Latest Stories
CI/CD is conceptually straightforward, yet often technically intricate to implement since it requires time and opportunities to develop intimate understanding on not only DevOps processes and operations, but likely product integrations with multiple platforms. This session intends to bridge the gap by offering an intense learning experience while witnessing the processes and operations to build from zero to a simple, yet functional CI/CD pipeline integrated with Jenkins, Github, Docker and Azure...
"We do one of the best file systems in the world. We learned how to deal with Big Data many years ago and we implemented this knowledge into our software," explained Jakub Ratajczak, Business Development Manager at MooseFS, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Today, we have more data to manage than ever. We also have better algorithms that help us access our data faster. Cloud is the driving force behind many of the data warehouse advancements we have enjoyed in recent years. But what are the best practices for storing data in the cloud for machine learning and data science applications?
The hierarchical architecture that distributes "compute" within the network specially at the edge can enable new services by harnessing emerging technologies. But Edge-Compute comes at increased cost that needs to be managed and potentially augmented by creative architecture solutions as there will always a catching-up with the capacity demands. Processing power in smartphones has enhanced YoY and there is increasingly spare compute capacity that can be potentially pooled. Uber has successfully ...
All zSystem customers have a significant new business opportunity to extend their reach to new customers and markets with new applications and services, and to improve the experience of existing customers. This can be achieved by exposing existing z assets (which have been developed over time) as APIs for accessing Systems of Record, while leveraging mobile and cloud capabilities with new Systems of Engagement applications. In this session, we will explore business drivers with new Node.js apps ...
Using new techniques of information modeling, indexing, and processing, new cloud-based systems can support cloud-based workloads previously not possible for high-throughput insurance, banking, and case-based applications. In his session at 18th Cloud Expo, John Newton, CTO, Founder and Chairman of Alfresco, described how to scale cloud-based content management repositories to store, manage, and retrieve billions of documents and related information with fast and linear scalability. He addres...
The technologies behind big data and cloud computing are converging quickly, offering businesses new capabilities for fast, easy, wide-ranging access to data. However, to capitalize on the cost-efficiencies and time-to-value opportunities of analytics in the cloud, big data and cloud technologies must be integrated and managed properly. Pythian's Director of Big Data and Data Science, Danil Zburivsky will explore: The main technology components and best practices being deployed to take advantage...
For years the world's most security-focused and distributed organizations - banks, military/defense agencies, global enterprises - have sought to adopt cloud technologies that can reduce costs, future-proof against data growth, and improve user productivity. The challenges of cloud transformation for these kinds of secure organizations have centered around data security, migration from legacy systems, and performance. In our presentation, we will discuss the notion that cloud computing, properl...
Chris Matthieu is the President & CEO of Computes, inc. He brings 30 years of experience in development and launches of disruptive technologies to create new market opportunities as well as enhance enterprise product portfolios with emerging technologies. His most recent venture was Octoblu, a cross-protocol Internet of Things (IoT) mesh network platform, acquired by Citrix. Prior to co-founding Octoblu, Chris was founder of Nodester, an open-source Node.JS PaaS which was acquired by AppFog and ...
By 2021, 500 million sensors are set to be deployed worldwide, nearly 40x as many as exist today. In order to scale fast and keep pace with industry growth, the team at Unacast turned to the public cloud to build the world's largest location data platform with optimal scalability, minimal DevOps, and maximum flexibility. Drawing from his experience with the Google Cloud Platform, VP of Engineering Andreas Heim will speak to the architecture of Unacast's platform and developer-focused processes.
The deluge of IoT sensor data collected from connected devices and the powerful AI required to make that data actionable are giving rise to a hybrid ecosystem in which cloud, on-prem and edge processes become interweaved. Attendees will learn how emerging composable infrastructure solutions deliver the adaptive architecture needed to manage this new data reality. Machine learning algorithms can better anticipate data storms and automate resources to support surges, including fully scalable GPU-c...
The vast majority of businesses now use cloud services, yet many still struggle with realizing the full potential of their IT investments. In particular, small and medium-sized businesses (SMBs) lack the internal IT staff and expertise to fully move to and manage workloads in public cloud environments. Speaker Todd Schwartz will help session attendees better navigate the complex cloud market and maximize their technical investments. The SkyKick co-founder and co-CEO will share the biggest challe...
When applications are hosted on servers, they produce immense quantities of logging data. Quality engineers should verify that apps are producing log data that is existent, correct, consumable, and complete. Otherwise, apps in production are not easily monitored, have issues that are difficult to detect, and cannot be corrected quickly. Tom Chavez presents the four steps that quality engineers should include in every test plan for apps that produce log output or other machine data. Learn the ste...
With more than 30 Kubernetes solutions in the marketplace, it's tempting to think Kubernetes and the vendor ecosystem has solved the problem of operationalizing containers at scale or of automatically managing the elasticity of the underlying infrastructure that these solutions need to be truly scalable. Far from it. There are at least six major pain points that companies experience when they try to deploy and run Kubernetes in their complex environments. In this presentation, the speaker will d...
While some developers care passionately about how data centers and clouds are architected, for most, it is only the end result that matters. To the majority of companies, technology exists to solve a business problem, and only delivers value when it is solving that problem. 2017 brings the mainstream adoption of containers for production workloads. In his session at 21st Cloud Expo, Ben McCormack, VP of Operations at Evernote, discussed how data centers of the future will be managed, how the p...