Telstra exchange fire spread by cooling systems

Telstra will spend over $10 million restoring its damaged Warrnambool exchange after revealing how fire, smoke and melted plastic spread via air-conditioners, in-rack fan cooling systems and propped-open fire doors.

Melted cable trays in the Warrnambool exchange (courtesy: Telstra)

The carrier today released its technical investigation into the exchange fire on November 22 last year, which destroyed about 60 percent of the facility and led to service outages of up to 20 days.

Though the report (pdf) sheds little light on the ignition source — thought to be some sort of electrical fault in a room for office space and storage of spare parts — it provides extensive clues on how the fire and associated damage spread, and 22 recommendations to prevent a recurrence.

It reveals that air-conditioning systems in the building continued to operate on standby power for at least 33 minutes after the fire started.

"Soot and burnt plastic cabling particulates were distributed throughout the exchange by the operating air conditioning and permeated equipment racks, particularly those with integrated fan cooling systems," the report states.

"This resulted in many hundreds of printed circuit boards failing due to the chemical residues or overheating due to fan failure or equipment air filters clogging up."

Though airflow helped the fire along, so did open smoke doors in the maintenance control room, in or above which the fire is thought to have started.

Smoke doors are meant to double as fire doors and stop smoke from filling escape routes. Although they aren't meant to completely stop the spread of flames, they are meant to be self-closing.

"Smoke doors of the maintenance control room were left open and allowed the fire to spread," Telstra noted, recommending that all Telstra employees and contractors be subjected to a "general awareness campaign ... relating to the requirement to close smoke and fire doors".

The Warrnambool exchange had heat, smoke and Very Early Smoke Detection Aspirated (VESDA) detectors fitted elsewhere in the facility that all worked as expected.

The report notes there are about 200 sites Australia-wide that house similar levels of telecommunications infrastructure to that of the Warrnambool exchange.

All are to undergo a building inspection using thermal imaging equipment to uncover any "hot spots", and have smoke and heat detectors fitted in ceiling spaces.

The exchanges will see other improvements stemming from the report's 22 recommendations, at an additional cost to Telstra.

"The total cost in terms of when the fire initially started through to completing all the permanent restoration works ... will be a little over $10 million for Warrnambool," chief operating officer Brendan Riley told iTnews.

"We've got that budgeted in the current financial year for Telstra.

"Obviously the 22 recommendations will apply to other similar exchanges around Australia, and we'll budget to undertake some of that work between now and June, and more of it into the next financial year."

Big red buttons

Recommendations have been made to allow quick override of mains and standby power sources to prevent them continuing to feed air conditioning systems, and for clear instructions to be available to the fire brigade to operate isolate power supplies.

"Firemen isolated the site from the AC supply and disabled the [standby generator plant], however the DC power systems continued to supply the load which saw the smoke and soot drawn into sensitive equipment by DC operated fan units," Telstra stated.

"The damage to telecommunication equipment would have been lessened if the DC power supplies were quickly isolated.

"All [exchange] sites in regional Australia with multiple DC power systems should have clear instructions for local Fire Brigade staff to be able to isolate by operation of circuit breakers the telecommunication load from the DC Power system in the event of a fire."

Riley said the carrier would make it simpler for firefighters to shut all power to an exchange facility.

"If you look at when the fire brigade came into the Warrnambool exchange one of the first things they did is switch off the power, which you obviously need to do," he said.

"What that did is it started our backup diesel generators, which is what they were designed to do. So then [the firefighters] had to shut those down.

"Then once they'd done that then they also had to switch off the DC power, since exchanges really run off batteries topped off by AC [power].

"We've got to put some big red buttons in there that just make it easy for the fire brigade, update our procedures, update the simulation and disaster recovery plans that we're working with the fire brigades, and then review our air conditioning policy."

Internal cabling

Another recommendation is for cable path diversity in exchange sites, to prevent swathes of critical cables from sitting in a single cable tray (and melting at once).

"Within Warrnambool the optical cables connecting the external cables to the external trunk transmission equipment were all in the same cable trays and all extensively damaged by the fire," Telstra stated.

Riley indicated lack of cable tray diversity is a problem in other exchange buildings, and would be among the first permanent repairs made to the Warrnambool site.

Telstra's investigation also revealed an absence of "fire pillows" — fire retardant fabric bags — in some of the exchange's cable trays, which also did not help in nullifying the fire.

"We've got to look to how we can build more cable tray diversity in the exchanges and also make sure that we've got the fire pillows accurately embedded in those trays," Riley said.

"In the case of Warrnambool that will be one of the first things we change, we will physically separate the fibre optic cables from the traditional copper cables. They'll be on different sides of the exchange.

"Then we'll have a lot of work ahead of us on other major exchanges around the country."

Read on for the state of network config data stores and Telstra's response to how it communicated with the community about outages and restoration.

Routing data stores

The carrier also plans to create better offsite stores of network configuration data, such as exchange routing tables, to allow faster reloading and "the return to a previous known working network configuration".

Riley told iTnews that the files in use at the exchange had been recoverable, so backups were not required to be called on, but the state of the backups wasn't ideal.

"When you went and had a look at the backups they could have been better, quite frankly," he said.

Offsite storage of accurate backups could have enabled Telstra incident teams to get started remotely "a little earlier", Riley acknowledged.

"That in turn would have helped us speed the recovery," he said.

"Overall today [the backups are] satisfactory but I think there's a lot we can do to move them up a notch and make them much more immediately serviceable."

Where to from here?

Permanent repairs to the Warrnambool exchange were approved last month and are expected to be finished by the end of June this year.

Telstra's technical findings are also expected to feed into a separate federal inquiry that deals more with the social and community impacts of the exchange fire.

iTnews has previously reported early submissions from this inquiry, including estimates that the fire cost Victoria's south-west economy $408,000 a day, and that individual losses were above $10,000 in some cases.

Telstra expects to begin addressing community communication impacts from April, according to Riley.

"I think what we'll be kicking off after April is a review of the communication and our processes," he said, referring to the release of today's technical investigation as "part one" of a broader probe.

However, he believed Telstra had consistently communicated its 3+3+3 disaster recovery methodology, which outlines roughly when communities can expect to see services repaired or permanently restored.

That methodology has full permanent restoration after three months, a target Telstra won't hit in this instance as it put tools down between December 2012 and February 2013.

"That's primarily because we wanted to shut down over that Christmas holiday period and not make any changes," Riley said.

He does see room for improvement in the way Telstra communicated some aspects of the recovery efforts, and remains apologetic for all affected customers.

"I think we could have been a little more forthright with that communication out into the community," he said, noting any findings would also be handed over to the federal inquiry.

The federal inquiry is due to release its findings some time this year.

Riley said he wanted to acknowledge the substantial efforts of the Country Fire Authority, which has so far prepared two reports of its own on the incident, and also the "men and women of Telstra" that had fed the recovery effort.

"This is a one in 50-year event," Riley said.

"The last exchange fire we had was in 1961 in Canberra, and I just want to commend [Telstra staff] on the speed of recovery because I think it would have to go down as a world record for the rebuild of an exchange anywhere."