Cloudflare reveals 'bad' data leakage bug

Content delivery network and security provider Cloudflare has owned up to a "bad bug" that was leaking sensitive information, after being alerted to the problem by Google Project Zero researcher Tavis Ormandy.

Cloudflare today said it was contacted by Ormandy last Friday to report a problem with the company's edge servers.

Ormandy spotted "corrupted web pages being returned by some HTTP requests run through Cloudflare", company programmer John Graham-Cumming said.

"It turned out that in some unusual circumstances ... our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data," he wrote.

"And some of that data had been cached by search engines."

Graham-Cumming said customer SSL private keys had not been leaked, as Cloudflare terminates SSL connections through an isolated Nginx instance that was not affected by the bug.

The company quickly turned off its email obfuscation, server-side excludes, and automatic HTTPs rewites features, which were using the HTML parser chain that caused the leakage, stopping memory from being returned in an HTTP response.

The parser used by the three features was written by Cloudflare last year, and named cf-html. It was intended to replace the Ragel parser that had "become too complex to maintain". The company has been slowly migrating functionality using the Ragel parser to cf-html.

"It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal Nginx buffers were used," Graham-Cumming said.

"Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself."

The bug was caused by a coding error that resulted in a buffer overrun. The problem had existed for years but only manifested itself once Cloudflare began migrating away from the Ragel parser.

"Our internal infosec team is now undertaking a project to fuzz older software looking for potential other security problems," Graham-Cumming said.

Once Cloudlfare was alerted to the problem, teams were established across departments and the globe to investigate the cause of the bug, and to work with search engine operators to remove any cached HTTP responses, he said.

The bug was fixed globally in under seven hours as a result of the 24/7 global effort, he claimed.

"We are disclosing this problem now as we are satisfied that search engine caches have now been cleared of sensitive information," Graham-Cumming said.

"We are grateful that it was found by one of the world’s top security research teams and reported to us."

Graham-Cumming said there was no evidence that the bug had been exploited.

Around one in every 3.3 million HTTP requests sent through Cloudflare from February 13-18 potentially resulted in memory leakage, he revealed.

However, Ormandy said Cloudflare had been leaking customer HTTPS sessions - from the likes of Uber, 1Password, FitBit, and OKCupid - for "months".

"[Cloudflare] worded [their notification] confusingly. It was exploitable for months, we have the cached data," Ormandy said.