Stemming Heartbleed with the human element

One of the largest security stories in 2014 was the extremely well-publicised Heartbleed vulnerability, which affected a very popular piece of software that nobody knew much about: the open source OpenSSL cryptographic library.

Tim Hudson

It was a very simple bug, according to the OpenSSL project’s Tim Hudson. The code in OpenSSL didn’t check how long a buffer was, a flaw that could be abused to silently leech information from vulnerable servers.

Because it was packaged and presented with a logo, a website and layman explanations, Heartbleed received a massive amount of coverage, even in mainstream media.

How did it happen?

Heartbleed occurred because the OpenSSL source code was complex, arcane and hard to maintain, which in turn made it difficult for developers to contribute to it, Hudson told iTnews ahead of his presentation at AusCERT 2016 today.

The main OpenSSL developers were overworked and overcommitted. There were no documented policies, and the project itself had become somewhat moribund; all this contributed to Heartbleed lying unnoticed in the source code for three years, Hudson said.

UNIX purists in the OpenBSD operating system project - which emphasises correctness and security - had had enough, and forked OpenSSL into LibreSSL to clean up the code.

In 2014, the OpenSSL project was in crisis.

But in 2016, OpenSSL is a totally different beast. The project earnt a security best practices badge from the Linux Foundation’s Core Infrastructure Initiative this month.

“The latest round of badges includes an assessment of OpenSSL, the open source software responsible for most encryption on the internet, both before the Heartbleed vulnerability and after it received support from CII. Prior to Heartbleed, OpenSSL failed to meet more than one-third of the CII best practices badge criteria. Today it meets 100 percent,” the CII said.

So how did Hudson and others fix OpenSSL?

“Simple answer: more resources, more structure, more focus. Basically sorting out the human resource element for the project was the single most important outcome in addressing Heartbleed,” Hudson said.

This may be hard to believe, but even though OpenSSL started in 1998, most team members had never met each other in real life.

“It did indeed take Heartbleed to get a physical face to face meeting for the team - the majority of the team members had never even met up with each other, or with more than one other team member,” Hudson said.

“Most had no idea as to what the other members involved were like people outside of the email exchanges.

“We plan to have another face to face meeting in Europe later this year."

But there’s more to the human element in development than meeting up. You have to understand people, Hudson said, and what happens when they’re under pressure.

“The most simple thing - which applies to all human activity not just software development - people make mistakes, busy people make more mistakes," he said.

"You not only have to have experienced people reviewing code, you have to be sure they are actually reviewing it and that there are multiple reviewers - or the frequency of mistakes will be higher than what you want."

Having got back on track with the coding, the next step for OpenSSL is to get the Federal Information Processing Standards 140-2 validation. Being FIPS 140-2 validated is a mandatory requirement for vendors to the United States government agencies, and it’s an important goal for the OpenSSL project.

But Hudson conceded achieving FIPS 140-2 validation will be an uphill struggle.

“The on-going FIPS 140 validation is a challenge - a couple of vendors have indicated an interest - but there is nothing sorted out as yet - as it is a substantial commitment of time and money - and there is no guarantee of success,” he said.

“Unlike other vendor validations absolutely everything is visible, which enables anyone that has an issue with OpenSSL for whatever reason to lodge an objection.

“It means that OpenSSL itself has a lot of additional effort required because of everything being open. But that is perhaps an entirely different story for another time.”