Failover fumble behind ASX Trade outages

The Australian Securities Exchange (ASX) has revealed this week’s outages were the result of sub-systems that failed to connect to a back-up copy of the ASX Trade database.

In a full post-incident report, the ASX said a hardware failure in the Australian Liquidity Centre early on Monday September 19 triggered failover procedures to a back-up copy of its main database at a separate, undisclosed site.

While the database failed over correctly, sub-systems that connected into it appear to have been misconfigured, and continued to try to connect to the broken primary database.

The ASX said that while its technicians were initially unaware of the “anomaly”, it was identified prior to the normal market opening time of 10am, but fixing it manually led to the later open of 11.30am.

“The sub-systems that comprise ASX Trade and that utilise the database are designed to automatically reconnect to the new primary database,” the ASX said.

“This reconnection process was only partially successful. A number of sub-systems incorrectly maintained connection to the failed database. This malfunction created a series of technical issues.

“The impact of the failed reconnection of the common sub-systems was visible to [market] participants as unsuccessful log-ons due to incomplete queries.

“ASX failed over these common sub-systems to force a reconnection to the new primary database and opened the market at 11.30am.”

However, not all securities were available to trade at 11.30am; two “groups” were forced to open later at midday and 1.20pm AEST respectively. The ASX blamed an “operational error".

Not long after at 1.42pm AEST, a “reference data inconsistency” forced the ASX to pause the market again. It would not re-open for the rest of the day.

ASX closures are a rare event; prior to this week, the exchange said it had been forced to do so only three times in the past decade.

While the closures were "extremely disappointing", CEO Dominic Stevens sought to justify the exchange's decision.

"ASX does not decide to close the market lightly," he said in a statement.

"Understandably, concerns have been raised on various fronts.

"However, we make no apologies for following our long-established and well understood protocols, nor for exercising our judgement to ensure the market was fair, orderly and transparent.

"That is the ASX's responsibility."

The ASX had earlier blamed the outages on a hardware failure, however this was challenged by industry experts.

The technical issues caused problems for both the ASX and Chi-X Australia trading venues.