Internode finds bug in Ericsson DSLAMs

ISP Internode has discovered a software flaw that has crippled some of the Ericsson-branded Ethernet controller nodes (ECNs) on its network - a flaw that may lie dormant within the networks of other service providers.

Internode's managing director Simon Hackett posted a lengthy technical explanation of the issue late last week, which had caused broadband service issues in some exchanges in South Australia.

Hackett said the fault had been "traced to a software bug in one of the multiple software systems running in the ECNs."

He said the faulty software release had operated "flawlessly for around five months until this fault developed" and said there had been "zero indication of the potential for this to happen".

"The current theory, based on vendor [Ericsson's] feedback, is that the software fault is a latent, load triggered, memory leak that makes the ECN lose the plot in a manner that corrupts some critical item inside one of the ECN systems - an item that remains corrupted after it is rebooted or power cycled," Hackett said.

"Our vendor has subsequently identified a bug that is consistent with our observed outcome, and on their advice, we upgraded to the next revision of the code concerned (in which that specific bug had been fixed)."

The newer software release had resolved issues for customers on affected ECNs. However, Hackett said customers on yet-to-be-upgraded switches were also becoming affected.

"Based on vendor advice, we are now looking at how best to upgrade the rest of the network, to guard against a further recurrence elsewhere," he said.

iiNet saved by earlier upgrade

It was understood that at least one other major Australian ISP - Perth's iiNet - had the same model Ericsson ECNs in its ADSL2+ DSLAM configuration.

iiNet chief technology officer Greg Bader told iTnews they had already upgraded affected boxes.

"We upgraded ours earlier in the year - [we] saw some issues similar to Internode but not really much customer impact," Bader said.

"We had a failure rate of around 0.05 percent with the ECNs, [which meant] minimal customer impact."

It was unclear whether customers that used the affected software were notified to upgrade their systems. An Ericsson Australia spokesman has been contacted for comment.

Ericsson counts ISPs including iiNet, Internode, TransACT, Primus, TSN, Netspace and Adam Internet among its DSLAM customers.

Some are understood to run networks based on a different model of Ericsson DSLAM, which did not run the same software.

It was not the first time that a software bug had impacted Ericsson DSLAMs.

In 2007, iiNet ADSL2+ users experienced connection problems using the point-to-point protocol over Ethernet (PPPoE) method that was caused by faulty software.