Backup bug caused Nasdaq outage

By on
Backup bug caused Nasdaq outage

Failover failed to work.

A massive trading halt at the Nasdaq last week was due to a software bug triggered by problems at the New York Stock Exchange's Euronext's Arca exchange that led a key backup system to fail, the exchange operator revealed.

Nasdaq said it was "deeply disappointed" by the three-hour outage on August 22, and while it pointed to connection problems between rival NYSE's electronic exchange Arca and the Nasdaq-run system that receives all traffic on quotes and orders for Nasdaq stocks, it took ultimate responsibility for the glitch.

"Our backup system did not work," Bob Greifeld, Nasdaq's chief executive, said.

"There was a bug in the system, it didn't fail over properly, and we need to work hard to make sure it doesn't happen again," he said.

NYSE declined to comment.

Nasdaq said it was in the process of identifying potential design changes to make the Securities Information Processor, or SIP, more resilient, "including architectural improvements, information security, disaster recovery plans and capacity parameters."

The exchange plans to present its initial recommendations for change to the SIP governing committee, made up of US exchanges and the Financial Industry Regulatory Authority, within 30 days.

Nasdaq said on the morning of August 22, Arca connected and disconnected to the SIP more than 20 times, eating up capacity. It said the SIP's capacity was further eroded as Arca sent a stream of inaccurate stock symbols to the SIP, generating numerous rejection messages.

Each data port connecting to the SIP can handle 10,000 messages per second, Nasdaq said. But in this case, the traffic from Arca was more than double that, the exchange said.

"The confluence of these events vastly exceeded the SIP's planned capacity, which caused its failure and then revealed a latent flaw in the SIP's software code," the Nasdaq report said.

This software flaw prevented the processor's built-in backup system from resetting properly, delaying the return of data. With the system degraded, the exchange said it decided to halt trading to ensure fair market conditions.

Nasdaq said it took 30 minutes to resolve the problem and then nearly three more hours to test and evaluate scenarios to reopen the market in a fair and orderly manner.

"Nasdaq OMX is deeply disappointed in the events of August 22 and our performance is unacceptable to our members, issuers and the investing public," the exchange said.

"While getting to 100 percent performance in all of our activities, including our technology is difficult - it is our objective."

Got a news tip for our journalists? Share it with us anonymously here.

Most Read Articles

Log In

Username / Email:
  |  Forgot your password?