Writing up a post-incident report to customers is a position few IT project leaders want to find themselves.
The support desk is swamped with tickets, many of which are lodged by the same customers, for the same issue. Those with direct lines into senior IT staff are bypassing that mess altogether.
Everyone wants answers. What's the root cause? Who (or what) caused services to be offline?
If the root of the issue traces back to your vendor, contractor or outsourced third-party, tread carefully.
The rules of scapegoating are unwritten and the game changes from case to case.
When a storage area network failure in 2009 led to an email outage affecting 200,000 customers, WebCentral steadfastly refused to 'out' the vendor whose array failed.
An outage at Queensland's Polaris data centre in March this year was blamed on an "equipment vendor"; again, unnamed.
However, Queensland Health blamed integrator IBM within a day of a highly critical audit of its bungled payroll system.
IBRS advisor Rob Mackinnon offers a rule of thumb when considering the apportionment of public blame.
"There's an old saying that I used to use with my troops - when you start pointing the finger, if you look at your hand there's actually three fingers pointing back at you," he says.
Cherub Consulting director John Liburti says that while vendors are at fault at times, "you probably want to look into your own house first before you start throwing those stones".
"It's convenient sometimes to be able to point at the third party and say, 'Look, we have an arrangement in place, we have a contract in place'. They were obliged to deliver to these requirements," he says.
"But the hardcore reality is that you [the company] have been placed in the position of authority to deliver the service. The vendor ... didn't authorise the project."
There are other reasons not to out your vendor publicly. Doing so could be symptomatic not just of a due diligence failure on your part but also of a relationship breakdown, which could determine whether the project is delivered at all.
If the relationship is indeed beyond repair, public shaming may also impede any hope of securing legal restitution or damages.
"If your project ain't going to work then ... get some money for it," Mackinnon says.
"The best chances of getting ... compensation for a failed project is by keeping it quiet."
On the chin
For problematic IT projects, the consensus is that the buck stops internally.
"The reality of the matter is, irrespective of who's at fault, the company takes the hit," Liburti says.
"One way or another, they are going to be in the firing line from their customers because it was their decision that resulted in the outsource in the first place."
Blaming a vendor or third-party ignores internal responsibilities and could point to failures in project planning and due diligence.
"It goes back further than due diligence," Mackinnon says.
"It comes down to clearly articulating what it is you're trying to do, what the role of the vendor is going to be, going through the process of selecting that vendor, then doing the due diligence as to whether that's an appropriate vendor."
Liburti says that due diligence processes should flag potential problems with vendors before they occur.
Even when statements of work and expectations are well-defined and understood, some vendors may attempt to sign a deal "on the understanding or belief that they could put in the adjustment" to the contract later, he warns.
"A good company will prevent that from happening," he says.
Another red flag could be contract acceptance with few or no objections.
"When you get back a response from a vendor and there's nothing ticked as a standout item in the contract, they agree to all the services and the service levels, and they'll deliver all the reporting as required, that may very well be the case," Liburti says.
"But buyer beware - let's ask the question. Let's do our due diligence and just find out, particularly if you have some comparative information - [for example], other vendors have responded against some liability clauses, yet this particular vendor hasn't.
"There must be a reason for that. Don't assume that it could have been missed by the legal department or it could very well have been deliberately avoided.
"Signing them up ... knowing full well they may very well fail ... is just as bad as not telling them in the first place that there was a liability."
He adds: "You need to look inward before you start looking outward. It's not always the vendor that's at fault. Don't necessarily start throwing rocks."
Make it work
Another reason not to publicly shame a vendor or supplier is to not jeopardise the investment that has been pumped into the project to date.
"Generally there's multi-millions of dollars worth of investments and benefits that are down the end of the pipeline, hopefully if the thing works," Mackinnon says.
"As a former CIO, I'd be looking at getting the vendor into the camp with me, working in partnership and making the damn thing work.
"I'd be working really really hard to get the solution to work until it looked like it was an impossibility [because] you are duty-bound within your organisation to make it work, and if it isn't going to work then to gracefully walk away."
Read on to page two for practical examples of when to 'out' a vendor and tips for dealing with vendor meltdown.