Telstra exchange fire spread by cooling systems

Telstra will spend over $10 million restoring its damaged Warrnambool exchange after revealing how fire, smoke and melted plastic spread via air-conditioners, in-rack fan cooling systems and propped-open fire doors.

Melted cable trays in the Warrnambool exchange (courtesy: Telstra)

The carrier today released its technical investigation into the exchange fire on November 22 last year, which destroyed about 60 percent of the facility and led to service outages of up to 20 days.

Though the report (pdf) sheds little light on the ignition source — thought to be some sort of electrical fault in a room for office space and storage of spare parts — it provides extensive clues on how the fire and associated damage spread, and 22 recommendations to prevent a recurrence.

It reveals that air-conditioning systems in the building continued to operate on standby power for at least 33 minutes after the fire started.

"Soot and burnt plastic cabling particulates were distributed throughout the exchange by the operating air conditioning and permeated equipment racks, particularly those with integrated fan cooling systems," the report states.

"This resulted in many hundreds of printed circuit boards failing due to the chemical residues or overheating due to fan failure or equipment air filters clogging up."

Though airflow helped the fire along, so did open smoke doors in the maintenance control room, in or above which the fire is thought to have started.

Smoke doors are meant to double as fire doors and stop smoke from filling escape routes. Although they aren't meant to completely stop the spread of flames, they are meant to be self-closing.

"Smoke doors of the maintenance control room were left open and allowed the fire to spread," Telstra noted, recommending that all Telstra employees and contractors be subjected to a "general awareness campaign ... relating to the requirement to close smoke and fire doors".

The Warrnambool exchange had heat, smoke and Very Early Smoke Detection Aspirated (VESDA) detectors fitted elsewhere in the facility that all worked as expected.

The report notes there are about 200 sites Australia-wide that house similar levels of telecommunications infrastructure to that of the Warrnambool exchange.

All are to undergo a building inspection using thermal imaging equipment to uncover any "hot spots", and have smoke and heat detectors fitted in ceiling spaces.

The exchanges will see other improvements stemming from the report's 22 recommendations, at an additional cost to Telstra.

"The total cost in terms of when the fire initially started through to completing all the permanent restoration works ... will be a little over $10 million for Warrnambool," chief operating officer Brendan Riley told iTnews.

"We've got that budgeted in the current financial year for Telstra.

"Obviously the 22 recommendations will apply to other similar exchanges around Australia, and we'll budget to undertake some of that work between now and June, and more of it into the next financial year."

Big red buttons

Recommendations have been made to allow quick override of mains and standby power sources to prevent them continuing to feed air conditioning systems, and for clear instructions to be available to the fire brigade to operate isolate power supplies.

"Firemen isolated the site from the AC supply and disabled the [standby generator plant], however the DC power systems continued to supply the load which saw the smoke and soot drawn into sensitive equipment by DC operated fan units," Telstra stated.

"The damage to telecommunication equipment would have been lessened if the DC power supplies were quickly isolated.

"All [exchange] sites in regional Australia with multiple DC power systems should have clear instructions for local Fire Brigade staff to be able to isolate by operation of circuit breakers the telecommunication load from the DC Power system in the event of a fire."

Riley said the carrier would make it simpler for firefighters to shut all power to an exchange facility.

"If you look at when the fire brigade came into the Warrnambool exchange one of the first things they did is switch off the power, which you obviously need to do," he said.

"What that did is it started our backup diesel generators, which is what they were designed to do. So then [the firefighters] had to shut those down.

"Then once they'd done that then they also had to switch off the DC power, since exchanges really run off batteries topped off by AC [power].

"We've got to put some big red buttons in there that just make it easy for the fire brigade, update our procedures, update the simulation and disaster recovery plans that we're working with the fire brigades, and then review our air conditioning policy."

Internal cabling

Another recommendation is for cable path diversity in exchange sites, to prevent swathes of critical cables from sitting in a single cable tray (and melting at once).

"Within Warrnambool the optical cables connecting the external cables to the external trunk transmission equipment were all in the same cable trays and all extensively damaged by the fire," Telstra stated.

Riley indicated lack of cable tray diversity is a problem in other exchange buildings, and would be among the first permanent repairs made to the Warrnambool site.

Telstra's investigation also revealed an absence of "fire pillows" — fire retardant fabric bags — in some of the exchange's cable trays, which also did not help in nullifying the fire.

"We've got to look to how we can build more cable tray diversity in the exchanges and also make sure that we've got the fire pillows accurately embedded in those trays," Riley said.

"In the case of Warrnambool that will be one of the first things we change, we will physically separate the fibre optic cables from the traditional copper cables. They'll be on different sides of the exchange.

"Then we'll have a lot of work ahead of us on other major exchanges around the country."

Read on for the state of network config data stores and Telstra's response to how it communicated with the community about outages and restoration.