Having trouble sending email?

Thousands of Australian businesses have been unable to send emails this week due to a configuration error in a blacklisting service operated by Cisco Systems.

The issue, which has baffled IT administrators all week, incorrectly gave a large set of IP addresses – many of them Australian customers of Cisco’s IronPort service – a poor reputation score.

Many organisations use Cisco’s IronPort web reputation service (known as SenderBase) to determine whether to accept emails from a given IP address, as a means to cut down on spam.

Business users whose emails have not reached their destination have turned to network managers, ISPs and web hosts in frustration to seek answers.

A technical advisory issued by Canberra-based hosting company AussieHQ suggests legitimate mail has been blocked since as far back as Saturday. The web host released a service advisory Saturday noting “clients may be experiencing bouncebacks when sending through our IronPort mail system.”

By Tuesday it became clear that the problem was affecting mail services across the nation, including customers of ISP Internode.

John Lindsay, carrier relations manager for the telco, told iTnews the issue was "a great example of over-blocking, and one of the reasons why automated filtering of websites on a great scale doesn’t work".

"Internode runs about a million mailboxes and at the end of the day someone, somewhere around the world is going to see something they don’t like," he said.

"We’ve seen that happen to Optus, iiNet and various other mail servers all the time and it’s something that actually I think doesn’t work that well which is why we spend a lot of money on industrial-strength mail filters that actually make their own decisions on the likelihood of it being spam and so forth."

Network managers running the Catholic Education Network in South Australia noted on Twitter that “much of Internode’s address space seems to be getting a bad mail reputation,” preventing Internode customers from sending mail to recipients on the CESA network. “Senderbase.org seems to have applied a 'guilty by association' policy for email traffic from network blocks with poor reputation.”

It took conversations between this journalist and two resellers in Victoria and South Australia to get to the root of the problem.

It appears that Cisco made some aggressive configuration changes to the IronPort system at some stage late last week, causing numerous false positives.

This publication has learned Cisco Systems has acknowledged the fault internally and told customers to wait 24 hours until a fix updates on IronPort servers. It remains unclear how much longer afterwards customers will have to wait for their reputation score on Senderbase will return to normal.

Customers appear to be especially vulnerable if they have multiple reverse pointer records for a single IP address. This is a common scenario, for example, when there are multiple hosts on a shared web server.

It is also commonplace for an ISP, rather than the customer, to control part of the naming process and the server administrator the other – leading to multiple names for a single IP address.

“We have never bothered getting them updated previously,” one server administrator told iTnews. “It looks like now we will have to.”

Precisely how many email users are impacted is difficult to determine. The spread of affected users in discussion with iTnews to date suggests that it is nationwide, but one reseller said the issue “will have a significant impact on the internet worldwide.”

Representatives from Cisco Systems and AussieHQ have been asked to respond to the story, but were unable to before going to press.

James Hutchinson contributed to this report.

Have you or your end users had emails fail to arrive at the destination this week? Let us know below. We’ll keep you updated throughout the day.