Easing The Pain of Being On-Call

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

Every time the phone rang in the middle of the night I pretty much knew that some soulless voice would be on the line. The monotonous robotic voice would state something like “JOB J-S-1-5-3 ABENDED AT LINE NUMBER 12 WITH ERROR CODE 1234.”

For the thousands of IT staff that are “on-call,” this is a familiar, and very much unwelcome, experience. Over the years, systems support staff have seen their sleep interrupted by beepers, Blackberry vibrations, and cell phone calls. I guess this beats having the home phone ring and wake up the spouse and kids. It is hard enough getting yourself out of dreamland and back to the cold “back-to-work” mentality in the middle of the night.

I remember the first time I received a beeper to be on-call in the early 90’s. I felt so important! Whenever I was at a party I made sure to wear my beeper proudly on my belt, never tucked away in my pocket. And when it went off, it was with great pride I would proclaim just loud enough for all to hear, “Excuse me but I need to call into the computer center.”

Smart IT Columns

Understanding Your ‘Idiot’ Manager

Is Everyone Replaceable?

Managing Vendor Demonstrations

Blogging Your Way Up the Career Ladder

That got old very, very fast.

When the beeper started going off in the middle of dinner, during a crucial point in my favorite TV show (this was pre DVR), and repeatedly in the middle of the night, I started to think being on-call wasn’t a very glamorous responsibility. As a matter of fact, it was the pits. And sometimes after working into the wee hours of the morning to solve a problem, I had to be back at work very early writing more code while feeling groggy and irritable. I began to wonder how the heck we could do more to reduce the number of calls or at least make the life of on-call staff a bit less stressful.

It should be quite obvious that better coding standards and techniques will reduce the number of bugs. And you don’t have to work in a CMM or ISO9001 environment to have processes in place. However, in many IT shops the staffing levels still haven’t recovered from the post-bubble staff reductions and it is human nature to look for time saving short cuts when we are overworked. It is also true in small to mid-size businesses that there isn’t a dedicated support staff and that every team member must take a turn at support or be in the escalation hierarchy.

So I get that everyone is stretched. However, to reduce your late night calls, you must not only put standards and processes into place, but put in a mechanism to enforce them. I’m a big fan of peer-to-peer, standards-based, code and unit test reviews – without a manager in the room. Teams can do a very good job of policing themselves, and with more vocal participation. Once code passes this review, then passes quality assurance and integration testing, it can be blessed for a promotion to your production environment.

A developer should never be able to promote their code into production without following this process – except while on call. If they have to fix something to get the system running again, then consider it a temporary patch that must then go through the same review and testing process, preferably the next day.

At least today remote access is fast and easy. Back in the Stone Age, dialing into the mainframe on a 2400-baud line was very time consuming. Make sure your team has the tools in place at home to remotely diagnose a problem, such as Citrix Presentation Server and a high speed Internet connection. Don’t ask them to rely on their home computers for support. You don’t want their teenager to download a virus on a computer they need for support. Either assign them a laptop or have one that is shared by the support team.

Next page: More Solutions for On-Call Help

There are also less tangible things you can do to make your on-call folks more productive (and less cranky). Try to share the burden, instead of having a dedicated support person. If you put your entire development and QA team into rotation, there will be a smaller chunk of the year spent on-call for each of them. These are the experts because they write and test the code, so if something goes wrong they can probably fix it faster. If a developer puts in a new module, then adjust the schedule so that person is on-call the first time it runs.

Be consistent when dealing with holiday support schedules. If someone was on-call on Thanksgiving, then don’t assign them this duty again for Christmas. Announce next year’s annual schedule before this year’s December holiday season so people have a chance to review it.

Smart IT Columns

Understanding Your ‘Idiot’ Manager

Is Everyone Replaceable?

Managing Vendor Demonstrations

Blogging Your Way Up the Career Ladder

Have a process in place for people to swap on-call slots. You can always veto a swap if you feel a person’s expertise is needed for a particular slot. Just make amends for any veto with extra time off or some other perk.

Be extra sensitive to the impact being on-call has to life outside work. When you are on-call, you really can’t mentally leave work completely. You have to always make yourself available if trouble occurs and thus have to be careful not to make extravagant plans (or consume alcoholic beverages at the local bar).

Just watching little kids while your spouse goes out can become a nightmare if you get a support call. Consider providing a half-day off the Monday after on-call ends. This will give them a few extra hours of family time or just extra time to decompress. Even if they received no calls, they still had to be on alert, which increases stress levels.

In the extreme cases of a team member being up late (or early) working on a problem, be more flexible in their working hours the next day. Do you think the code I wrote under sleep deprivation was solid? If they are up many nights in a row, consider giving a comp day.

By improving your processes to reduce bugs, providing the support tools, and being more sensitive to the stresses of round-the-clock support, you will create an environment that eases the pain of being on-call.