Welcome!

Blog Feed Post

Trainmageddon: When the machines stop working, people get upset.

For the train companies of the United Kingdom, today was a tough one. How’s this for a headline (which we all gazed at during our morning coffee break):

If you haven’t heard, you can read all about how the poor train companies of the UK copped a battering by commuters on social and traditional media for a ticket machine malfunction. I feel for the commuters but my sympathy today lies with the train companies. Well except for the ticket collectors that obviously didn’t get the memo, and handed out fines to those that boarded without a ticket!

Coverage:

Here’s why it’s so hard to be consistently perfect

Purchasing a single ticket is actually a hyper complex transaction from an IT point of view.

The complexity links from the (and let’s list them which isn’t even all of them!):

  1. front end software the customer uses at the attempted purchase
  2. third party payment gateway that processes the payment
  3. integration between the machine (of the device), the software and the gateway
  4. security certificate required to make a secure transaction
  5. credit check application
  6. hosting environment in which all this runs
  7. interconnection between the transactions that are crossing different hosting environments, from the end user, the train station, and through to the back end applications.

Yet all the customer cares about is their experience at the machine – it needs to be perfect, or if there’s a problem, it needs to be resolved in seconds…so they can board their train on time.

Not like this:

So what happened today in the UK?

We may not find out for sure what happened but from our experience, monitoring millions, if not billions of transactions a day, there are three common areas where problems can arise. When it comes to IT complexity, rapid release cycles and digital experience, typically problems centre on:

Human error – Oops did I do that? 

Software needs updating. Software updates are mostly written by humans, and when multiple humans are working together, it’s not uncommon for mistakes to be made. Even if you have the most stringent pre-production testing, issues can still arise once you push to production because you can never accurately replicate what software will do in the wild.

As our champion devops guru Andreas Grabner always preaches in his talks – #failfast. If the issue relates to change that was made, roll it back, fast.

in this case with the outage today, I doubt it was a software update in the core operating system. I’d expect it was a third party failure, which incidentally might have had it’s own update. But more on this on point 3.

Security

Not one to speculate on, but obviously when a software failure causes mass disruption to people, it would be fairly normal to assume that maybe some sort of planned security attack. But again I doubt it.

Delivery chain failure

The most likely cause for the train machine failure is simply a failure somewhere in the digital delivery chain. Considering a single transaction today runs across 82 different technologies, from devices, networks, 3rd party software applications, hosting environments, and operating systems, it doesn’t take much for a single failure to cause a complete outage. Understanding where that is, so that you can quickly resolve is critical. Referencing what I said in point 1, it’s probable that a simple update to any of these 82 different technologies caused a break in the chain. Or maybe one of these 3rd parties had their own outage.

And that’s where AI comes in.

This is why you need AI powered application monitoring, with the ability to see the entire transaction across every single one of the different technologies. But not just across the transaction, but the ability to go deep from the end point machine, to the host infrastructure, the line of code, and the interconnections between all the services and processes.  It’s the only way you can identify the root cause of the problem – in minutes not hours, or days.

The days of eye balling charts, having war room discussions with IT teams, are definitely over. Software rules our lives, and it simply cannot fail. Otherwise digital businesses face a day like this on social media:

What if the machine fixed the machine?

With the ability to see the immediate root cause of a problem, it’s not improbable for the machine to learn how to course correct itself. In the same way when servers are overloaded, a load balancer can direct traffic to a under utilised host. So if you can detect an issue in the delivery chain then the machine can about self correcting itself with an alternative path.  If the payment gateway fails, then it could auto redirect to a new hosted payment gateway for instance. Our chief technical strategist Alois Reitbauer demo’d just this scenario (ok a more simpler version) at Perform 2017. So it’s not that far off.

The post Trainmageddon: When the machines stop working, people get upset. appeared first on Dynatrace blog – monitoring redefined.

Read the original blog entry...

More Stories By Dynatrace Blog

Building a revolutionary approach to software performance monitoring takes an extraordinary team. With decades of combined experience and an impressive history of disruptive innovation, that’s exactly what we ruxit has.

Get to know ruxit, and get to know the future of data analytics.

Latest Stories
"I think DevOps is now a rambunctious teenager – it’s starting to get a mind of its own, wanting to get its own things but it still needs some adult supervision," explained Thomas Hooker, VP of marketing at CollabNet, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We are still a relatively small software house and we are focusing on certain industries like FinTech, med tech, energy and utilities. We help our customers with their digital transformation," noted Piotr Stawinski, Founder and CEO of EARP Integration, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We've been engaging with a lot of customers including Panasonic, we've been involved with Cisco and now we're working with the U.S. government - the Department of Homeland Security," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We're here to tell the world about our cloud-scale infrastructure that we have at Juniper combined with the world-class security that we put into the cloud," explained Lisa Guess, VP of Systems Engineering at Juniper Networks, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"With Digital Experience Monitoring what used to be a simple visit to a web page has exploded into app on phones, data from social media feeds, competitive benchmarking - these are all components that are only available because of some type of digital asset," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, provided a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services with...
"Peak 10 is a hybrid infrastructure provider across the nation. We are in the thick of things when it comes to hybrid IT," explained Michael Fuhrman, Chief Technology Officer at Peak 10, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
As enterprise cloud becomes the norm, businesses and government programs must address compounded regulatory compliance related to data privacy and information protection. The most recent, Controlled Unclassified Information and the EU’s GDPR have board level implications and companies still struggle with demonstrating due diligence. Developers and DevOps leaders, as part of the pre-planning process and the associated supply chain, could benefit from updating their code libraries and design by in...
SYS-CON Events announced today that Calligo, an innovative cloud service provider offering mid-sized companies the highest levels of data privacy and security, has been named "Bronze Sponsor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalised support service from its globally located cloud plat...
"We are an IT services solution provider and we sell software to support those solutions. Our focus and key areas are around security, enterprise monitoring, and continuous delivery optimization," noted John Balsavage, President of A&I Solutions, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We were founded in 2003 and the way we were founded was about good backup and good disaster recovery for our clients, and for the last 20 years we've been pretty consistent with that," noted Marc Malafronte, Territory Manager at StorageCraft, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol,...
"We are focused on SAP running in the clouds, to make this super easy because we believe in the tremendous value of those powerful worlds - SAP and the cloud," explained Frank Stienhans, CTO of Ocean9, Inc., in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"DivvyCloud as a company set out to help customers automate solutions to the most common cloud problems," noted Jeremy Snyder, VP of Business Development at DivvyCloud, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"At the keynote this morning we spoke about the value proposition of Nutanix, of having a DevOps culture and a mindset, and the business outcomes of achieving agility and scale, which everybody here is trying to accomplish," noted Mark Lavi, DevOps Solution Architect at Nutanix, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.