Cisco says mea culpa on bounced emails

Claims problem fixed.

Cisco Systems claims to have remediated the issue at the heart of widespread email failures for businesses in Australia and globally in recent days.

As first reported by iTnews, a reconfiguration in the company's IronPort SenderBase service (set to be rebranded as Cisco Security Intelligence Operations) caused it to erroneously block outbound email for business users across Australia since Monday at the earliest.

Cisco Systems today told iTnews that the problem was global and affected a much wider base than Australia.

The SenderBase service takes in 35 percent of the world’s email traffic – some 100 million messages a day, chewing through four terabytes of data daily from 750,000 endpoints across Cisco’s customer base.

The company first detected the problem Tuesday morning in California, or Monday night in Australia.

“Cisco became aware
 of an issue that resulted in Cisco email security products blocking some
 legitimate email senders,” the company said in an emailed statement.

“This was the result of recent algorithm 
updates that focused on newly identified internet traffic behaviors
 indicating spam activity. Unfortunately, this traffic behavior is also
 seen in some legitimate email activity, resulting in the inaccurate 
blocks.

”

Glenn Welby, manager of the IronPort portfolio in the Asia Pacific region, explained that Cisco hires some 500 employees to monitor changing patterns in email behaviour and identify shifts in the way criminals attempt to extort money or send spam.

At times these staff adjust the 200-odd elements that contribute to the web reputation algorithm accordingly.

“Our staff noted a particular change in the way criminal behaviour was taking place and changed the algorithm to diminish the reputation of email sent under those conditions,” he said, declining to disclose the criminal activity under investigation.

“Candidly, we made a mistake and impacted some legitimate users.”

Cisco’s statement suggests the algorithm was remediated by Tuesday evening on the US West Coast, or Wednesday morning in Australia.

But SenderBase profiles showed Australian customers were still impacted as late as Thursday morning Australian time.

“In our 'follow-the-sun' model of tech support, the caseload began in the United States, has moved over to Australia today and will move to Europe when the Australian team goes to bed,” Welby said.

Welby said the company is still undertaking root-cause analysis for an issue that is “only 24 hrs old in US terms".

He said it would be premature for network and server administrators to make changes to their domain pointer structure to remediate the problem, as suggested in iTnews’ initial article.

“In [building] the elements that create a reputation and feeding it into SIO, we have a good understanding of what is legitimate and what is not legitimate activity. This was simply a mistake," he said.

“We’ve rectified that, and the issue is fixed for any end user who has IronPort in their environment.”

Welby said it would be disingenuous to nominate a date for when those blacklisted by SenderBase would have their reputation scores return to normal.