“We are brilliant in a crisis” is not the compliment you think

Being “brilliant in a crisis” is not the compliment it seems, because excelling at incident management usually means getting a great deal of practice fixing failures that should not have happened. John Sansbury argues that IT and Estates teams in higher education should shift effort from heroic recovery to prevention.

• Incident management is the only purely reactive process in service management; its sole benefit is restoring a service that should not have broken.
• Four barriers keep organisations reactive: everyone is too busy firefighting, unreasonable deadlines push teams to cut corners, prevention costs are visible while failure costs are spread invisibly, and a hero culture rewards the dramatic rescue over the uneventful year. The Millennium Bug shows that prevention is thankless precisely when it works best.
• The cost is large. Research cited in the book puts the cost of putting right what should not have gone wrong at up to 30 per cent of turnover, and outages at between 2,300 and more than 9,000 dollars per minute.
• The practical first step is problem management, which removes the causes of incidents rather than their symptoms. One trained problem manager who prevents a single serious outage a year pays for themselves several times over.
• That person must be dedicated. Assigned alongside a day job, they will be pulled back into firefighting, and recurring issues such as hall lift failures or annual timetabling clashes will simply recur.