Welcome!

Blog Feed Post

Stepping Up to the Plate: A Story About Being On-Call

The Alert

A few weeks ago, I went to my first baseball game. The San Francisco Giants were playing the San Diego Padres at the AT&T park, and my relatives had an extra ticket for me. I met my relatives at the front entrance of the park, and when we entered I took the whole spectacle in: the big LED lights, the endless rows, the infinite hallway of food stalls. After gathering the necessary garlic fries and chicken tenders, we made our way to our seats.

The Giants make three runs in the first inning, and the whole stadium was electric with excitement. But the sixth inning, the lead had flipped, and the Padres were beating the Giants by three runs. Tensions were high in the ballpark, with the few fans clad in blue getting louder and louder, while those sporting orange grew silent.

The Ghirardelli man was making his rounds, and my cousin flagged him down to grab us some hot chocolate. He makes his way to our row, pours a cup and passes it down to me. My hands grab the cup — and then my phone goes off. I’m startled by the loud ring and vibration, and the cup of hot chocolate slips from my hands. My cousin sitting beside me catches it, though my jeans take some stains. The spectators behind us complain, telling me to silence my phone. My phone was on silent though. I had configured it to only make a sound if it was an alert from PagerDuty.

“Hold that for me, there’s something I have to do,” I tell her.

“You okay? What’s wrong?” My cousin asks.

“There’s been an incident, I need to go.”

I grab my headphones from my purse, stand up, push my way past the legs of the seated spectators of row three, and run up the stairs.

The Response

I roamed around trying to find a private place to take the call, but everywhere I went the speakers blared and the cheers reverberated throughout the stadium. At the end of the food hall, I spot the illuminated sign and book it into the bathroom. The acoustics only amplified the crowd’s jeering, but I was running out of time. I pick the farthest stall from the entrance, put the toilet seat down, plug my headphones in and join the call. I mute my microphone, I did not want the background noise to bother anybody. I was the third person to join the call, and I entered mid-conversation.

“We’re waiting for the on-call member from the EM team,” a voice says.

“All right, who is the EM on-call?” another asks.

“I’m not sure. We’ll just wait and-” the first voice is interrupted.

“Hello?” A third voice.

“Hi there,” someone replies.

“Hi, this the EM on-call.”

“Hello, what’s the situation, and what’s your status on resolving it?”

“I already resolved it, but let me get on the portal to make sure everything’s okay.”

“What!” I yell in disbelief. I cover my mouth, then realized (with relief) that they could not hear me. It had only been two minutes since the initial alert was sent, and the on-call engineer had already solved the incident before joining the call. In the next few minutes, the three voices started rattling off numbers and analyzing metrics. While I had no idea what any of it meant, I took it from the calm tone of their voices and lack of swearing that we were out of any sort of trouble.

“Yeah, it’s back to normal now.”

“Awesome. Do you have any reason to think this will happen again?”

“No, I don’t think that this will come up again, but I will keep an eye out.”

“All right then. Well, thanks for handling this.”

“No problem, thank you everybody for being here. Goodbye.”

“Goodbye, have a good weekend.”

The conference call ends, and I look at my phone screen. 8 minutes and 38 seconds. 8 minutes to resolve an incident, or to talk about it anyway. I sat there in the bathroom stall, dumbfounded. I come out of the stall to wash my hands, and noticed in the mirror that I had not attended to the dark hot chocolate stains on my jeans.

As I start trying to wipe the splotches away, I realize how grossly under prepared I was for what had happened. I was stressed and flustered, and I was only shadowing. One, I did not have my laptop with me. Two, my phone was on 15% charge. Three, I had one too many beers I doubt I could have solved any sort of technical problem, let alone explain what I was solving to someone else. If I were the on-call engineer, I would have struck out. I would have let my team down.

The Post-Mortem

That evening, the Giants came back in the bottom of the ninth inning, and I realized that being on-call is somewhat like baseball. Specifically, being on-call is like being the batter when your team has two outs, has third base filled, and is down by one point at the bottom of the ninth inning. In that moment, the team’s success rides on you and you alone. In front of you, you have teammates on the bases, and their success is entirely dependent on yours. Behind you, you have the rest of the team in the dugout, waiting to see whether you fail or fly.

The batter swings and the ball is in play. That was when it clicked. With PagerDuty, the on-call engineer is no longer the lone batter, and is instead one of the players on the field. With PagerDuty, being on-call ceases to be an individual endeavor: it becomes a team sport. Instead of the on-call engineer having to sift through thousands of alerts to find the problem and solve it on his own, he had a team to support him, and a central line in which they could communicate, and a platform that filtered out all the unnecessary noise. When the ball is in play they assess the situation, they pass it around to who is best positioned to solve the problem, all with the common goal of resolving the issue before it shows up on the customer’s screen.

PagerDuty’s platform goes beyond making sure that the customer’s digital experience is seamless and smooth: it makes the on-call experience less stressful, less uncertain and less overwhelming.

I do not have a technical background in engineering or computer science, nor am I a huge sports fan, so I find it humorously ironic that I was able to make sense of both these things by putting them together.

The post Stepping Up to the Plate: A Story About Being On-Call appeared first on PagerDuty.

Read the original blog entry...

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

Latest Stories
What is the best strategy for selecting the right offshore company for your business? In his session at 21st Cloud Expo, Alan Winters, U.S. Head of Business Development at MobiDev, will discuss the things to look for - positive and negative - in evaluating your options. He will also discuss how to maximize productivity with your offshore developers. Before you start your search, clearly understand your business needs and how that impacts software choices.
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp em...
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.
Agile has finally jumped the technology shark, expanding outside the software world. Enterprises are now increasingly adopting Agile practices across their organizations in order to successfully navigate the disruptive waters that threaten to drown them. In our quest for establishing change as a core competency in our organizations, this business-centric notion of Agile is an essential component of Agile Digital Transformation. In the years since the publication of the Agile Manifesto, the conn...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
There is huge complexity in implementing a successful digital business that requires efficient on-premise and cloud back-end infrastructure, IT and Internet of Things (IoT) data, analytics, Machine Learning, Artificial Intelligence (AI) and Digital Applications. In the data center alone, there are physical and virtual infrastructures, multiple operating systems, multiple applications and new and emerging business and technological paradigms such as cloud computing and XaaS. And then there are pe...
Real IoT production deployments running at scale are collecting sensor data from hundreds / thousands / millions of devices. The goal is to take business-critical actions on the real-time data and find insights from stored datasets. In his session at @ThingsExpo, John Walicki, Watson IoT Developer Advocate at IBM Cloud, will provide a fast-paced developer journey that follows the IoT sensor data from generation, to edge gateway, to edge analytics, to encryption, to the IBM Bluemix cloud, to Wa...
SYS-CON Events announced today that Keisoku Research Consultant Co. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Keisoku Research Consultant, Co. offers research and consulting in a wide range of civil engineering-related fields from information construction to preservation of cultural properties. For more information, vi...
While some developers care passionately about how data centers and clouds are architected, for most, it is only the end result that matters. To the majority of companies, technology exists to solve a business problem, and only delivers value when it is solving that problem. 2017 brings the mainstream adoption of containers for production workloads. In his session at 21st Cloud Expo, Ben McCormack, VP of Operations at Evernote, will discuss how data centers of the future will be managed, how th...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
Most of the time there is a lot of work involved to move to the cloud, and most of that isn't really related to AWS or Azure or Google Cloud. Before we talk about public cloud vendors and DevOps tools, there are usually several technical and non-technical challenges that are connected to it and that every company needs to solve to move to the cloud. In his session at 21st Cloud Expo, Stefano Bellasio, CEO and founder of Cloud Academy Inc., will discuss what the tools, disciplines, and cultural...
SYS-CON Events announced today that Fusic will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Fusic Co. provides mocks as virtual IoT devices. You can customize mocks, and get any amount of data at any time in your test. For more information, visit https://fusic.co.jp/english/.
SYS-CON Events announced today that Enroute Lab will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enroute Lab is an industrial design, research and development company of unmanned robotic vehicle system. For more information, please visit http://elab.co.jp/.
SYS-CON Events announced today that Mobile Create USA will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Mobile Create USA Inc. is an MVNO-based business model that uses portable communication devices and cellular-based infrastructure in the development, sales, operation and mobile communications systems incorporating GPS capabi...
With the rise of DevOps, containers are at the brink of becoming a pervasive technology in Enterprise IT to accelerate application delivery for the business. When it comes to adopting containers in the enterprise, security is the highest adoption barrier. Is your organization ready to address the security risks with containers for your DevOps environment? In his session at @DevOpsSummit at 21st Cloud Expo, Chris Van Tuin, Chief Technologist, NA West at Red Hat, will discuss: The top security r...