When working for a major retailer the on-call was supposed to be either a text or call from the service desk referencing an incident which required immediate attention. But quite often someone would call the service desk and instead of opening an incident they would simply ask for the phone number of the Linux on-call and would call us directly. On one occasion I had a DBA do this and call me on a Saturday morning for an issue in a non-prod environment. I would have something similar happen to me a few months later as I was the secondary on-call and the primary on-call failed to answer his phone. Needless to say this was a point of contention with our team and we pushed our management to get it changed.

We did have some success in having our management push back on the service desk to not do this and that they should be the only direct contact with the on-call person. Most of the other admins on the team took that as the opportunity to set their phones to only alert on calls and texts from the service desk number.

Fast forward to around 2:00 AM one Sunday morning and after a water pipe had broken over the server closet at one of the retail locations and work was underway to restore service. And though our team didn’t directly support the servers in question, the person from the backup team was having trouble restoring some files and needed some Linux assistance. They attempted to call the primary on-call…no answer. Then they tried the secondary…again no answer. They then escalated to our manager and funnily enough, he didn’t answer either. The next stop, our director and she answered the phone. After being apprised of the situation what was her response? “Call Todd, he’ll answer his phone.” And that’s what happened. I groggily answered and after being told I had been called I got online and provided the assistance.

---

Filed under: On-Call Stories - @ February 5, 2022 6:41 pm