How a US agency overhauled untenable security

When a US court ordered IT staff at the Department of the Interior to disconnect 85,000 staff from the internet in 2001, heads rolled.

It took four days to comply with the order, initially made in an attempt to keep hackers from accessing $US1 billion ($AU923.5 million) in Native American royalties managed by the department.

The decision came as a double blow to the department, which the same year was given a security score of 12 out of 100 by the US Congress - a "very very low F-grade", according to the man hired to clean it up.

It took the next six years and a federal court defence from a former chemical engineer and coal mine inspector Hord Tipton to convert one of the US' oldest civilian agencies into one of its most secure.

He ultimately saved hundreds of millions of dollars but the job wasn't easy.

The department remained embroiled in the longest-running legal fight in US history, a 149-year case with Native American communities who argued the government had squandered $uUS137 billion from the trust fund over more than a century.

The communities - and the US District Court - feared a further billion could be lost if the department's IT infrastructure wasn't tied down.

The plaintiffs asked that a penetration test be conducted on the systems of the Bureau of Indian Affairs. It was granted.

"They walked in the front door," Tipton told SC on a lightning trip to Sydney.

Of course, it would have been easy. Despite a multi-million dollar IT security budget, the bureau and wider department lacked even basic security structures. No firewalls or anti-virus applications.

"These people in the bureau were scientists, and they demanded unfettered internet access," Tipton explained. "And that meant no firewalls."

The District Court ruled that internet be severed to not just the offending bureau, but the entire department.

For bureau heads, it might have initially come as a relief; no more online distractions for its employees. But, as Tipton told it, a subsequent fortnight delay in delivering tens of thousands of pay checks to employees was only the start of a long line of problems.

The clean up

The Department of the Interior, known as 'the department of everything', manages over 500 million acres of government-owned land equal to approximately one fifth of the country's land mass.

More than a quarter of the nation's electricity is produced on land and seas managed by the department. It overseas oil reserves and the great Yellowstone National Park; some 500 dams including Hoover Dam and icons such as the Washington Monument.

Its networks and security controls were similarly disparate.

When Tipton stepped into the job, the department used 18 operating systems, 14 web portal solutions, 35 gateways, 153 financial payment systems, separate development systems for each bureau and kilometres of un-used fibre and telephone lines.

Tipton took to the IT cleanup operation wielding an indomitable razor.

He began with the department's new relationship database, dubbed 'ALMERS', that had failed after 15 years of development at a cost of $US10 million a year. Any changes to the system required code to be re-written.

"It used to take five minutes and cost about $10 to get a license to cut your own Christmas tree. After ALMERS, it took four hours and cost US$75," Tipton said.

It was promptly "taken out the back and shot".

A single Microsoft contract, replacing scores of disparate operating system and software licenses, saved $US40 million in the first year alone.

The department's 35 gateways were slashed to two (saving $US100 million) and disused networking lines, described as "T1s to nowhere", were consolidated to save $US500 million.

Disobedient bureaus that shunned the shared infrastructure model had their budgets cut. One agency that spent $US100,000 on building a duplicate in-house system had the same amount slashed from its coffers and redirected to the department.

The lesson was learnt quickly, and agencies soon fell into line.

He estimated that the clean up saved about $US150 million on information security spending alone.

Security overhaul

The court-ordered internet blackout allowed agencies to light back up as they improved information security. Tipton installed 252 point-to-point network links and "workarounds" between offices to keep the department operational.

He also forced the department's 225 information security staff to become Certified Information Systems Security Professionals (CISSP), or "get used to counting cattle".

Tipton himself passed after four months of cramming, despite not having a background in information security.

Incredibly, the offending bureau had successfully argued remain online, regardless of upgrades to its security.

The same court that berated the department was later impressed with information security overhaul, and lifted the internet blackout.

A department which once earned a government security rate of 12 receive a radically improved score of 79 out of 100 four years later. Tipton had successfully led it to become the first civilian agency in the country to receive the top rating for data resiliency.

Life lessons

Tipton left the department in 2007 to become head of security standards group ISC². While he said security standards in the department had since waned - with four CIOs leaving in as many years - he believed he learnt valuable lessons in security management.

"You can have the best security policy in the word but it won't work unless staff are on board and do it for themselves," Tipton said.

He called the rejection of policy controls "the silent veto" and said that technical systems must "demand compliance" to be successful. But they must also stand-up for questioning, and security chiefs need to be able to justify policy requirements to disgruntled staff.

The experience taught him an "appreciation for security" and he now carries the belief that all organisations should consider that data breaches are inevitable.

To this end, he recommended information security professionals consider their breach contingency plans and thoroughly test the resilience of their systems.

"We are in a different world now, after the September 11 attacks. Attackers are looking to fool your staff with clever phishing attacks, social engineering, and are still successful in breaching through simple exploits."

Copyright © SC Magazine, Australia