Blog Feed Post

BabyDuty or: How PagerDuty Accidentally Prepared me for Fatherhood

Becoming a new parent has been one of the most difficult challenges I’ve ever faced. That wasn’t a huge surprise; I’d been fully expecting it. But what I did find surprising was that my experiences with PagerDuty helped prepare me in ways I never anticipated. The following highlights some of my adventures in fatherhood so far, and how years of working as an on-call engineer has given me some much needed assistance during these first few months.


https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_1-300x215.jpg 300w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_1-1024x732... 1024w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_1-250x179.jpg 250w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_1-180x129.jpg 180w" sizes="(max-width: 632px) 100vw, 632px" />

Rich with his daughter, the new BabyDuty “service”.

It’s worth noting that this is a little tongue-in-cheek, and that I’m not really an emotionless robot who sees my daughter as a PagerDuty service… Not yet anyway.

PagerDuty ALERT: Labor is starting!

When my wife and I found out that we were pregnant (let’s be honest though, one of us was more pregnant than the other), one of my concerns was that when labor started, I would be somehow stuck down a mine shaft without a cell signal and would have no way of knowing. This feeling was exacerbated by the fact we’d just moved to a new city and my office was no longer a 10 minute drive from home.

I wanted to be absolutely sure that I didn’t miss a message from my wife telling me labor had started, so PagerDuty was an obvious solution. All I had to do was create a new PagerDuty service, give my wife the contact details, and set the escalation policy to page me via every possible notification method — every minute, on a loop — until I acknowledge. Unsurprisingly, this works really well!

As with all good alert management mechanisms, something I’ve learned the hard way is that you need to make sure to test that it works too. I would occasionally ask my wife to test it, putting my mind at ease that everything was still set up correctly. Knowing that we had this set up helped to keep me calm and prepared in the weeks leading up to the big day. This kind of preparation would only help in the days and weeks to come.


It turns out the PagerDuty alert we set up for labor has actually come in useful after my daughter was born too.

Picture the scene. It’s the middle of the night of our second week home, the perfect time to get paged. My daughter has woken up and it’s my turn to change and feed her. Unfortunately, I’m a heavy sleeper, and a baby’s cries don’t really wake me, nor does the nudging of my wife. One thing definitely does wake me though: my PagerDuty notification sound! It’s a custom notification sound I’ve used for the last 15 years of being on-call, and it is a unique ringtone that no one else has. Merely hearing that sound sends me from deep sleep to adrenaline fueled awakeness in a matter of microseconds.

Whenever my wife needs my help, whether it’s day or night, she knows that she can page me and I’ll always respond in seconds. We’ve used this in all sorts of situations, such as when my wife was trapped with a sleeping baby in her arms and needed me to bring her food and water while I was working in the yard. She could page me with a single touch of her phone rather than trying to type a message or call to me and potentially wake our sleeping baby.

It’s also been useful as a second alarm clock. If I’m so exhausted that I slept through my normal alarm, my wife who’s already up with the baby sometimes pages me letting me know I’ll be late for work otherwise.

I’m still working on putting some of my hard-earned automation skills to good use by wiring up a baby monitor to trigger PagerDuty alerts, thus skipping the need for my wife to manually page me. Keep an eye out for the “Baby Pager 5000”, coming soon.

Incident Commander-ing Labor and Delivery.

One of my roles at PagerDuty is that of an Incident Commander. I’ve been heavily involved in PagerDuty’s incident response process since I joined the company, and earlier this year I open-sourced our incident response documentation to share some of the things we’ve learned. I also started, and continue to run, our incident commander training sessions.

Having this kind of incident response experience has been invaluable in my life outside of PagerDuty in ways I never would have expected. I used to get easily frustrated if I couldn’t solve problems quickly, but I’ve started to notice that I’m now a lot calmer when faced with these types of issues, and work the problem instead. I find myself not as worried when new and unique problems pop up; I’m now able to approach the problem methodically rather than just panicking. I trace this all back to my incident response experience.

During labor there was a lot going on. My wife was hooked up to all sort of machines that go beep and bing; there were tubes, needles, and all sorts of things attached with sticky tape. A variety of different people kept coming and going from the room, and each giving different (and sometimes contradictory) instructions; “Count up to 10”, “Count down from 10”, “Push”, “Keep going”, “Breathe”, “Slower”, “Faster”. My wife also had worryingly high blood pressure during the entire process and required additional beeping machines and medications to be administered. To say it was a high-stress environment would be putting it mildly. But that made it all the more important to try and remain as calm as possible.

At some point, I ended up being the only one talking to my wife, repeating the important instructions with a calm cadence I never knew was possible in these moments. As incident commanders we’re taught to remain calm, provide clear and concise instructions, and to keep things progressing towards a resolution while ensuring others don’t get burnt out. All of this training came into play here, and the results were a huge success.


“I knew you were freaking out inside, but it really helped that you stayed so calm on the outside.”

– Rich’s Wife


My wife made a point to mention how calm I was throughout the whole experience and how it really helped her to stay calm and focused. I am absolutely sure that without my incident response training, I would not have had such a calm exterior. I can also reliably assure you that it was only a calm exterior. Anyone who says they weren’t at least mildly panicking on the inside during the birth of their child has to be lying, or a robot.

Incident response training has been applicable in many different ways since. Staying calm and focused was practical when driving our tiny baby home for the first time. Quickly sizing up a situation and prioritizing actions was significant when we ended up having to take our daughter back to the hospital less than a day after taking her home. It came up again when driving her home for the second time. Something I also learned is that babies can sense your moods, so if you get frustrated when trying to put them to sleep, they will get frustrated too. Staying calm in these moments is absolutely essential.

One final tip though: I wouldn’t recommend directly applying every aspect of incident response training to childbirth. For example, labor and delivery is not the right time to ask if there are any strong objections.

Scribing: Keeping track of everything.

Scribing during incident response is the process of keeping an accurate timeline of events and making notes of critical information and decisions. This ensures that everyone is on the same page and can review the key events after the fact.

In the first few weeks after your child is born, you have to keep track of all of their er… inputs and outputs. You need to keep a thorough log so that the doctors can see if anything is wrong. If they aren’t outputting often enough, it can be a sign of bigger issues. The problem is that you’re both exhausted. My wife had a real excuse; she’d just given birth. I had less of an excuse.

You don’t want to miss things because you’re tired. It’s very tempting during a feeding or diaper change to just say “Oh, I’ll log this later.” Just as it’s tempting to say “I’ll log this bug later” in software development. Don’t fall into the trap! We both made sure that we logged everything as it happened, without delay.

The result is a meticulous log of our daughter’s eating and expelling patterns, which has been invaluable. It’s incredibly useful to be able to share routines with caregivers, doctors, and each other. It also makes it very easy to fill in the countless wellness questionnaires your doctors will send.


https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_timeline-3... 300w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_timeline-2... 250w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_timeline-1... 180w" sizes="(max-width: 1024px) 100vw, 1024px" />

Tracking baby metrics, such as time feeding and sleeping, is invaluable for staying on top of things. This image shows our daughters sleeps (yellow) and feeds (blue) since she was born. Each vertical line represents a day going top to bottom.

We got so good at keeping the log that we’ve continued to do it to this day. It’s fantastic to be able to see the data change as our daughter’s sleeping patterns became more organized, and it provided a good motivational boost in those early weeks. As you can see, things really got better as time went on. As any operations team will tell you, the key is to track the metrics.


Monitoring metrics is only important if you can raise actionable alerts on those metrics when they don’t do what you expect.

While we track how many times a day our daughter has done her business, what happens if she hasn’t in awhile? Would we even notice? What’s great about tracking the metrics is that you can also set alerts for them. Which is exactly what we did!

If we haven’t logged a feed in over 5 hours, we have set a rule to page us both. If our baby hasn’t left us a gift in over 24 hours, same thing. Automating these alerts has taken a load off our minds, and helped to reduce our stress levels. It’s also been useful when I’m taking care of her to know when the last time she fed was, since I might not have been home at the time of her last feeding. It’s essentially automated an on-call shift handover.

Sometimes you freak out thinking that you’ve forgotten something important and you’re a horrible parent, but by putting in the time to pre-define some alerts, it’s taken away a huge source of stress, as we now know we’ve got an additional safety net. (Jury’s still out on if I’m a horrible parent. Ask again in a few years).

Not to mention all this will give me some great lines when my daughter is old enough: “You’ve been a pain in my ass since you were born and I have the metrics and graphs to prove it.”


https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_2-300x225.jpg 300w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_2-800x600.jpg 800w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_2-250x188.jpg 250w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_2-180x135.jpg 180w" sizes="(max-width: 693px) 100vw, 693px" />

 “Daddy, there’s no way I pooped that many times on Tuesday…”

Escalation is not always an option.

One thing that I’ve always had when on-call is a backup. A safety net. Knowing that if I can’t figure out a problem, I can call in someone to work it with me. When it comes to personal life, my wife is my backup, and I’m hers. We rely on each other to provide assistance and help whenever we need it.

But the problem now is that we’re both worn out most of the time. Four months in, our daughter went through a sleep regression. We were both physically and emotionally exhausted. It was my turn to put her back to sleep while my wife tried to sleep. Our daughter was crying and kicking because she was over-tired or just didn’t want to sleep.

I couldn’t escalate to my wife, since she had to get some much needed rest. There was no one else I could hand over to. I couldn’t just give up and call it a day. What can you do when escalation isn’t an option?

What it comes down to is the same for our Incident Commander process: keep calm, identify the symptoms, work the problem. If I’m on an incident call and am presented a problem without any backup, what do I do? Do I just give up and call it a day? No! I work the problem and figure it out. It might take longer, but the problem will get solved eventually.

In this case, I systematically worked through the things that could be wrong. Checked her diaper, tried to burp her, bounced her, walked around the room, shushed her with calming sounds, etc. Methodically work through each thing until you find something that works.

Maintaining sanity with effective on-call rotations.

Everyone told me before my daughter was born that the quicker you can get into a pattern, the better it’ll be. Easy for people to say, a lot harder to actually do it when you have a screaming two-week old in your arms. Any system you come up with gets thrown out of the window pretty quickly.

In the first few weeks, my wife and I worked in shifts so the other could eat and rest. This worked really well for a bit, until I had to return to work — completely throwing off our pattern.

But we came up with a new schedule and eventually got into a new pattern. This has become our on-call rotation, and has worked very well. Any time during our rotation, any incident (crying, etc.) that occurs is handled by the current on-call, while the other is free to do what they want.

This also comes in useful for other things such as conferences, friends visiting, dinners, etc. as we can just swap our on-call shift for another time. We follow the same patterns as we would if it were a real on-call shift.


Much to my disappointment, you are not provided with a simple runbook or flow chart for how to take care of a new baby (start-up idea!). The standard practice in cloud computing of resources being disposable doesn’t really apply here. You cannot reboot a baby. If a server is misbehaving, you can just terminate it and get a new one provisioned. You definitely can’t do that with a baby!

There are actually a vast number of resources available about how to take care of babies. Unfortunately, most of the information is conflicting and doesn’t really have much data or science backing it up. It’s like having a collection of out-of-date documentation. You don’t know which bits will work and which won’t.

But having a runbook is vital when dealing with the same situation over and over again. You can’t automate putting a newborn to sleep (another start-up idea!), but you can document a routine that works for you and make sure everyone who puts your baby to sleep follows the same procedure.

“We don’t like to hear it as operations engineers, but there are some things you really can’t automate. Putting a baby to sleep, for example.”

– Rich’s words of wisdom.

We tried out various routines for nap and bedtime, working through trial and error to find out a solution that got her to sleep effectively. Once we had a pattern that worked, we documented it with clear steps, providing instructions on what to do at each step if something didn’t work: “If she’s still fussy after 2 minutes, go to section 3”, etc. This has become our runbook, and has worked amazingly well.

It ensures that everyone who might be putting our daughter down for a nap or bedtime follows the same pattern, every time. It keeps things consistent for our daughter so that she knows what’s coming up, and means we get her to sleep with the minimal amount of fuss.

Documenting these types of routines has become useful for other things too. Checking weight and growth milestones follows a similar pattern. What to feed and when to feed, etc.

When you don’t have a runbook, you make your own based on past experience and build it up over time. Turns out it’s not much different with a baby. Just keep good documentation and learn from experience.

And just like with code, there’s no such thing as self-documenting babies.

Keeping your batteries charged.

Going on-call has always involved preparation. Specifically with making sure things are charged. A day before I go on-call, my routine is to make sure my laptop and mi-fi are both charging. I also make sure my phone backup batteries are fully charged, and I have one in each of my backpacks so I won’t get caught without them.

Whereas before with on-call I would have to keep my laptop with me at all times when out and about, it’s now a diaper bag. But the steps are remarkably similar. Before an outing, make sure supplies are properly stocked, and that you have extras for redundancy. Make sure I have multiples in various bags that I might be using. And finally, make sure I have my phone backup batteries fully charged. Because some things never change.

It’s OK to laugh.

Just like a major incident call, looking after a baby is a serious responsibility. It’s no time for jokes. You need to remain professional the entire time, never breaking from your seriousness. Except, that’s a complete lie.

A few weeks after our daughter was born, we were having trouble putting her to sleep. She was growing increasingly fussy, and nothing we tried seemed to work. My wife and I had swapped between ourselves about three times as we each got burnt out, and we were both close to breaking down. I cradled my daughter in my arms, and held her close to try and rock her to sleep. Then it happened. She obviously thought it was feeding time, but wasn’t aware that her father isn’t able to feed her in the way she wanted. She wasn’t going to let that stop her from trying though. My wife and I broke out in laughter, and in turn so did our daughter. The seriousness of the situation was broken and we just couldn’t stop laughing. After hours of having a fussy baby, a lighter mood was just what we needed to keep our sanity and turn a stressful environment into a relaxed one.

During times of high stress, taking a moment to laugh can be one of the best things for your mindset. Newcomers to PagerDuty are often surprised when joining an incident call to hear a joke every now and then, and people laughing. “Why are people laughing if there’s a critical incident in progress! Surely they should be solving the incident instead?” While moments should be chosen wisely, and there are certainly times when it would be inappropriate, I’ve found that injecting a joke now and then to lighten the mood actually improves the incident response process. Sometimes people just need a chance to relax a little, even in serious times.

Similarities and differences.

Despite all the things I’ve learned at PagerDuty and from previous engineering/on-call jobs that have helped with parenting, the one thing everyone always said was that it’ll be just like being on-call. It isn’t.

It’s harder.

Much harder.

In the weeks and months leading up to the birth of my daughter, pretty much everyone constantly wanted to remind me that I’m “obviously super prepared to be woken up in the middle of the night thanks to years of being on-call.” I’d just fake a smile and laugh. Turns out it doesn’t prepare you. Not even close.


“Hey, at least on-call will have you prepared you for the lack of sleep! Right? Right?!”

– Every single person I told that I was becoming a father


I’ve spent years being woken up by pages at 3am, glaring at a computer screen trying to fix an issue. Things are different when it’s a baby crying and screaming in your face. I’ve been on-call in the airline industry, where minutes matter, and one mistake in a weight and balance calculation can cause a plane to crash. That wasn’t as stressful as the first few weeks as a new parent.

But like on-call engineers, there’s one thing I can always count on: the compassion of others in the same position. As on-call engineers, we regale each other with stories of our worst and best times being on-call, with the problem we solved late at night in a moment of clarity. We have each other’s backs if we got hit with a particularly hard problem.

The support of other parents is something I’ve been overwhelmed with. The stories are different, the problems are similar, but the community element is something I would never take for granted.

Whether you’re a parent or not, there’s lessons to be learned and applied from the things you do. Incident response training is useful in many situations — whether it’s a production server exploding, a fender bender on the highway, or a midnight explosive diaper change. The lessons and tactics we use as engineers can be applied to situations we’ve probably never even considered.


https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_3-300x193.jpg 300w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_3-250x161.jpg 250w, https://www.pagerduty.com/wp-content/uploads/2017/06/babyduty_3-180x116.jpg 180w" sizes="(max-width: 713px) 100vw, 713px" />

“Daddy, I can’t even walk yet. Why are you using me for commercial branding? I better get royalties for this!”

The post BabyDuty or: How PagerDuty Accidentally Prepared me for Fatherhood appeared first on PagerDuty.

Read the original blog entry...

More Stories By PagerDuty Blog

PagerDuty’s operations performance platform helps companies increase reliability. By connecting people, systems and data in a single view, PagerDuty delivers visibility and actionable intelligence across global operations for effective incident resolution management. PagerDuty has over 100 platform partners, and is trusted by Fortune 500 companies and startups alike, including Microsoft, National Instruments, Electronic Arts, Adobe, Rackspace, Etsy, Square and Github.

Latest Stories
DX World EXPO, LLC, a Lighthouse Point, Florida-based startup trade show producer and the creator of "DXWorldEXPO® - Digital Transformation Conference & Expo" has announced its executive management team. The team is headed by Levent Selamoglu, who has been named CEO. "Now is the time for a truly global DX event, to bring together the leading minds from the technology world in a conversation about Digital Transformation," he said in making the announcement.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Conference Guru has been named “Media Sponsor” of the 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. A valuable conference experience generates new contacts, sales leads, potential strategic partners and potential investors; helps gather competitive intelligence and even provides inspiration for new products and services. Conference Guru works with conference organizers to pass great deals to gre...
DevOps is under attack because developers don’t want to mess with infrastructure. They will happily own their code into production, but want to use platforms instead of raw automation. That’s changing the landscape that we understand as DevOps with both architecture concepts (CloudNative) and process redefinition (SRE). Rob Hirschfeld’s recent work in Kubernetes operations has led to the conclusion that containers and related platforms have changed the way we should be thinking about DevOps and...
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develop...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
The next XaaS is CICDaaS. Why? Because CICD saves developers a huge amount of time. CD is an especially great option for projects that require multiple and frequent contributions to be integrated. But… securing CICD best practices is an emerging, essential, yet little understood practice for DevOps teams and their Cloud Service Providers. The only way to get CICD to work in a highly secure environment takes collaboration, patience and persistence. Building CICD in the cloud requires rigorous ar...
Companies are harnessing data in ways we once associated with science fiction. Analysts have access to a plethora of visualization and reporting tools, but considering the vast amount of data businesses collect and limitations of CPUs, end users are forced to design their structures and systems with limitations. Until now. As the cloud toolkit to analyze data has evolved, GPUs have stepped in to massively parallel SQL, visualization and machine learning.
"Evatronix provides design services to companies that need to integrate the IoT technology in their products but they don't necessarily have the expertise, knowledge and design team to do so," explained Adam Morawiec, VP of Business Development at Evatronix, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
"ZeroStack is a startup in Silicon Valley. We're solving a very interesting problem around bringing public cloud convenience with private cloud control for enterprises and mid-size companies," explained Kamesh Pemmaraju, VP of Product Management at ZeroStack, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Enterprises are adopting Kubernetes to accelerate the development and the delivery of cloud-native applications. However, sharing a Kubernetes cluster between members of the same team can be challenging. And, sharing clusters across multiple teams is even harder. Kubernetes offers several constructs to help implement segmentation and isolation. However, these primitives can be complex to understand and apply. As a result, it’s becoming common for enterprises to end up with several clusters. Thi...