Welcome!

Blog Feed Post

5 Things I Didn’t Expect to See While Shadowing On-Call

Shadowing on-call engineers feels like watching open-heart surgery over a doctor’s shoulder. Precision and speed are paramount, and mistakes can lead to severe consequences. As part of my, “Welcome-to-PagerDuty” experience, I was added to a PagerDuty engineering team’s on-call rotation as a shadow observer for a week.

During my time on-call, I was on the same escalation policies and received the same notifications as everyone else on the rotation. I would wake up if engineers on-call were woken up by a notification and lurk in the operations Slack channel as they would work through urgent time-sensitive situations. I expected to see stress bring out the worst in people. To my surprise, I was completely wrong.

Phones buzzed to interrupt sleep, meals, thoughts, and so much more — chat channels filled with rapid fire messages and resolution steps unfolded at all hours of the night. I was surprised to see kindness, support, and humor — despite the on-call responsibilities and stress.

Encouragement

https://www.pagerduty.com/wp-content/uploads/2017/09/shouldnt-be-hard-30... 300w, https://www.pagerduty.com/wp-content/uploads/2017/09/shouldnt-be-hard-25... 250w, https://www.pagerduty.com/wp-content/uploads/2017/09/shouldnt-be-hard-18... 180w" sizes="(max-width: 710px) 100vw, 710px" />

Source: https://xkcd.com/1349/

 

One simple sentence makes it easier to get through hours of stress.

In the late afternoon of my first-day shadowing, I got my first notification. I hopped on the Slack channel and saw that the investigation had already settled on diagnosing an issue from a recent deploy. The team immediately engaged, methodically pausing and provisioning new machines. In the flurry of triaging and diagnosing, I realized that on-call responder was as new to this situation as I was; we had both started less than one month ago. He was fixing a problem that he had never seen before.

In the early evening, things were getting back on track. It had been a long day and people were on their way home. Still, a veteran engineer (P) checked in and cheered on his teammate (K):

P [6:14 PM] Is everything done???

K[6:15 PM] Not yet, just in the middle of restarting the hosts. Everything is going smoothly so far though

P [6:18 PM] Way to go K! This sh*t used to take us half a day!

This type of peer encouragement creates confidence for people new to the project, new to the company, or new to the profession. Encouragement from senior engineers helps less experienced engineers feel brave enough to ask more questions. Asking questions is how new people learn and ramp up to become experts. It starts with empowering people: one “way to go” at a time.

(2) Humility and (3) Gratitude

The newbie on-call engineer in my anecdote immediately responded to the encouragement. He acknowledged and appreciated the senior engineers who had helped him. In three quick sentences, he demonstrated humility and gratitude.

K:[6:21 PM] No way. I would’ve sank if it weren’t for you and M. So much to learn!

I like that K called out another teammate who had helped him. DevOps engineering teams deeply value their teammates. More than just colleagues, they rely on each other to make sure that the software they’re writing today won’t create a hellish on-call rotation tomorrow. No one person can possibly know everything all the time; they have to constantly learn from each other and lean on each other for help.  

In the example above, observe the fine line between humility and self-deprecation. The engineer mentions that he has ‘so much to learn’. This attitude is different from ‘I’ll never do this myself’ or ‘I’ll never understand all of this stuff.’ Being humble and grateful builds trust between teammates. Self-deprecation can sow uncertainty and uneasiness.

 

(4) Levity and (5) So many gifs

https://www.pagerduty.com/wp-content/uploads/2017/09/alternate-currency-... 300w, https://www.pagerduty.com/wp-content/uploads/2017/09/alternate-currency-... 250w, https://www.pagerduty.com/wp-content/uploads/2017/09/alternate-currency-... 180w" sizes="(max-width: 433px) 100vw, 433px" />

Source: https://xkcd.com/512/

 

Being on-call can be repetitive and (blissfully) boring. On-call predictability reduces chaos and stress, but there will always be some inherent anxiety associated with being on-call. Luckily, the Internet granted on-call teams (and the rest of us) gifs to dissipate stressful on-call shifts with delightful bursts of humor on loop.

PagerDuty has a strong culture of humor and a particular passion for custom gifs. I loved observing this team, distributed across three separate time zones and two different countries, have fun and be goofy with each other. I noticed that people who weren’t currently on-call were consistently present and upbeat.

https://www.pagerduty.com/wp-content/uploads/2017/09/trustworthiness-274... 274w, https://www.pagerduty.com/wp-content/uploads/2017/09/trustworthiness-228... 228w, https://www.pagerduty.com/wp-content/uploads/2017/09/trustworthiness-164... 164w" sizes="(max-width: 433px) 100vw, 433px" />

Source: https://xkcd.com/1301/

 

So, how can you make your team’s on-call experience more positive?

  • Be present for your team, even if you’re not currently on-call
  • Cheer people on, especially people who are new to your team
  • Acknowledge when something is challenging, risky, or time-consuming
  • Be specific with your thanks — acknowledge the people who directly impacted you. It will help the team as a whole understand who has which strengths in future situations.
  • Be willing to get silly. Everyone loves to laugh.

I saw plenty of problem-solving, some heartache and frustration, and even parts of people’s weekends and evenings absorbed in being on-call. In the midst of it, small gestures from teammates remind you that you’re not alone.

When it feels like the sky is falling, it helps when people are excellent to each other.

For more on how to engender empathy in DevOps, check out our post on #HugOps.

The post 5 Things I Didn’t Expect to See While Shadowing On-Call appeared first on PagerDuty.

Read the original blog entry...

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

Latest Stories
SYS-CON Events announced today that Fusic will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Fusic Co. provides mocks as virtual IoT devices. You can customize mocks, and get any amount of data at any time in your test. For more information, visit https://fusic.co.jp/english/.
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
21st International Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Me...
SYS-CON Events announced today that Enroute Lab will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enroute Lab is an industrial design, research and development company of unmanned robotic vehicle system. For more information, please visit http://elab.co.jp/.
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
With the rise of DevOps, containers are at the brink of becoming a pervasive technology in Enterprise IT to accelerate application delivery for the business. When it comes to adopting containers in the enterprise, security is the highest adoption barrier. Is your organization ready to address the security risks with containers for your DevOps environment? In his session at @DevOpsSummit at 21st Cloud Expo, Chris Van Tuin, Chief Technologist, NA West at Red Hat, will discuss: The top security r...
IBM helps FinTechs and financial services companies build and monetize cognitive-enabled financial services apps quickly and at scale. Hosted on IBM Bluemix, IBM’s platform builds in customer insights, regulatory compliance analytics and security to help reduce development time and testing. In his session at 21st Cloud Expo, Lennart Frantzell, a Developer Advocate with IBM, will discuss how these tools simplify the time-consuming tasks of selection, mapping and data integration, allowing devel...
SYS-CON Events announced today that Mobile Create USA will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Mobile Create USA Inc. is an MVNO-based business model that uses portable communication devices and cellular-based infrastructure in the development, sales, operation and mobile communications systems incorporating GPS capabi...
Today traditional IT approaches leverage well-architected compute/networking domains to control what applications can access what data, and how. DevOps includes rapid application development/deployment leveraging concepts like containerization, third-party sourced applications and databases. Such applications need access to production data for its test and iteration cycles. Data Security? That sounds like a roadblock to DevOps vs. protecting the crown jewels to those in IT.
SYS-CON Events announced today that Interface Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Interface Corporation is a company developing, manufacturing and marketing high quality and wide variety of industrial computers and interface modules such as PCIs and PCI express. For more information, visit http://www.i...
SYS-CON Events announced today that Keisoku Research Consultant Co. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Keisoku Research Consultant, Co. offers research and consulting in a wide range of civil engineering-related fields from information construction to preservation of cultural properties. For more information, vi...
There is huge complexity in implementing a successful digital business that requires efficient on-premise and cloud back-end infrastructure, IT and Internet of Things (IoT) data, analytics, Machine Learning, Artificial Intelligence (AI) and Digital Applications. In the data center alone, there are physical and virtual infrastructures, multiple operating systems, multiple applications and new and emerging business and technological paradigms such as cloud computing and XaaS. And then there are pe...
SYS-CON Events announced today that SIGMA Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. uLaser flow inspection device from the Japanese top share to Global Standard! Then, make the best use of data to flip to next page. For more information, visit http://www.sigma-k.co.jp/en/.
SYS-CON Events announced today that B2Cloud will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. B2Cloud specializes in IoT devices for preventive and predictive maintenance in any kind of equipment retrieving data like Energy consumption, working time, temperature, humidity, pressure, etc.
Agile has finally jumped the technology shark, expanding outside the software world. Enterprises are now increasingly adopting Agile practices across their organizations in order to successfully navigate the disruptive waters that threaten to drown them. In our quest for establishing change as a core competency in our organizations, this business-centric notion of Agile is an essential component of Agile Digital Transformation. In the years since the publication of the Agile Manifesto, the conn...