Cisco fixes mystery orchestrator crashes

Enterprises using Cisco’s Network Services Orchestrator (NSO) software seeing mystery crashes now have a solution.

NSO is a multivendor system that links network automation and orchestration tools with the underlying physical and virtual infrastructure, with applications in enterprise and service provider environments.

In a recent field notice, the networking giant discovered a process in the software that causes a resources leak and eventual crash – and the time between crashes depends only on the resources allocated to the software.

It affects twenty versions of NSO 5, from 5.6 to 5.8.2.

The problem is that an API called cbd-diff-iterate is available to subscribers, but also used internally by Cisco for licensing and notification purposes.

“This affects customers who use Smart Licensing, use notification-kicker, or directly call the API through C, Python, Java, or econfd. 

“Any change under /kickers/notification-kicker or /devices/device will trigger an internal subscription, which leads to the leak”, Cisco’s field notice explained.

The internal subscribers cannot be disabled. 

“Once enough transactions have been performed, the Cisco NSO process will get terminated due to exhausting all available resources … The number of transactions that can be performed before termination will depend on the resources allocated to Cisco NSO,” Cisco explained.

Three versions have been released to fix the issue: NSO 5.6.7.1, 5.7.5.1, and 5.8.2.1.

For admins that can’t yet install the upgrades, Cisco recommends regular restarts.