'Firefighting' Means Trouble

Datamation content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

Have you ever noticed that some of the most respected IT people in a company are those that just dive into problems and fix them? These people repeatedly wade in, save the users, get a ton of accolades and everyone is happy. We term these people “firefighters” because they constantly move from one unplanned “fire,” or incident, to the next.

Paradoxically, organizations with lots of firefighting going on usually have unplanned work, high costs, unacceptable service levels, delayed projects and poor security. To say the least, this “firefighter” behavior and the culture that encourages it are problematic.

W. Edwards Deming, the noted quality advocate, used the analogy of a hotel being on fire. Someone who runs down the hall, sounds the fire alarm and then extinguishes the blaze would be rightly congratulated for a job well-done. But none of these heroics actually did anything to prevent future fires. Thus, people should move beyond dealing with fires and instead take action to improve the building, preventing fires from starting in the first place. And in IT, rushing from crisis to crisis is simply not the way to manage an enterprise.

As we all know, IT is a relatively immature profession with few, if any, standard processes and controls. As a result, IT, through human error, actually creates 80% of its own fires. That means that more than three-quarters of the problems are caused internally and are quite likely in IT’s realm of influence to prevent. Wouldn’t it be better to prevent fires in the first place and work on planned projects that would actually improve the organization? Of course it would — and here is how.

Regaining Control

First, put in a formal well-thought-out change management process. Use it initially as a last line of defense to catch mistakes before they go into production. By implementing this control gate, IT will have an opportunity to catch potential problems before they enter production.

Second, introduce detective controls to detect all changes — including ones that bypass change management. The detection and subsequent investigation of changes serve to drive the culture to change. As it is no longer possible to bypass change management, the firefighters must begin to follow the process.

Surprisingly, many of these people who resisted the change begin to see the benefits and become staunch advocates. People begin to understand cause and effect. The overall result is that changes are better planned, tested and deployed.This leads to a stabilized environment that has improved availability, more time spent on planned work and less firefighting.

If you need guidance on how change management works, there are three readily accessible resources. The Service Support volume of the ITIL book set can be purchased. It has excellent guidance on the overall practice. Additionally, the ITPI book Visible Ops: Starting ITIL in 4 Practical Steps uses a very intuitive four-phase approach for implementing change management and establishing a control environment. An additional reference guide is the Microsoft Operations Framework Service Management Function on change management.

For the naysayers who view change management as needlessly bureaucratic and/or as slowing the rate of change down too much, I recommend you take a serious look at how many of your changes are really solving problems versus creating more work. Look at your helpdesk logs and see how many calls and lost time could have been prevented if the changes had been better scrutinized. If you have a high rate of change, unacceptable levels of availability and a lot of projects delayed, never finished and/or never started, then you have a problem and must regain control.

In summary, firefighting in IT does have an occasional time and place where it is appropriate. But it should never be a standard way of doing business. IT must move beyond perpetual firefighting and focus on improvement. By implementing a well-thought-out change management process that uses detective controls, IT can stabilize the infrastructure and turn its focus to truly improving.