Why NHS contractors pushed patient data to the cloud

The UK’s National Health Service is at the centre of a fresh data privacy storm after a firm of technology consultants uploaded huge amounts of patient data to an offshore cloud in an effort to speed up an analytics processing job.

PA Consulting was contracted by the NHS to prove out a model for using hospital data to generate better patient outcomes for the UK health sector.

Under the agreement, PA purchased an archive of aggregated hospital activity across the UK, known as the Hospital Episode Statistics (HES).

The consulting firm installed the data on its hardware and uploaded it into a Microsoft SQL database on a 1TB server such that it could query the data to find out a range of information, such as admissions for particular conditions during the past decade.

PA's promotional materials, published online [pdf], noted that running these queries on its own hardware "still took several hours."

“With time and more storage space we could dramatically improve these times by normalising the tables and building suitable cubes. We could also load it into a more sophisticated analytics engine with dedicated hardware and software, but that would require significant capital investment.”

The consulting firm came up with a workaround: to upload HES to the Google Storage cloud and extract insights from it using Google's BigQuery analytics web service. PA found queries that took all night on its internal servers were returned in 30 seconds using cloud services.

Within two weeks of using the Google tools, PA was able to produce interactive maps from the data, each of which took mere seconds to generate. PA estimated that each of these maps would previously have taken months to produce. 

The company felt that this proved out the power of using a cloud service for batch analytics jobs.

“The slowness of the current process severely limits the quality and number of interesting results,” PA's brochure stated. “This means that analysis that should be highly influential in changing the way people allocate the NHS budget is not done at all or is too slow to have an effect.”  

The process could be improved without a large capital investment in technology, the firm noted.

PA Consulting has enjoyed kudos for its achievement, winning a UK IT Industry Award for best big data project and another from UK IT journal ComputerWeekly.

But links to these awards have since been pulled from the PA website after Conservative MP Sarah Wollaston, a member of the UK’s Health Select Committee, posted a tweet questioning whether any patients had consented to having their data uploaded.

@JuliaHCox @CLeslieMP so HES data uploaded to 'google's immense army of servers' , who consented to that @hscic ?!

— Sarah Wollaston MP (@drwollastonmp) March 3, 2014

Patient groups and privacy advocates are concerned that pushing sensitive information into the cloud leaves it open to abuse by insurance companies, government departments and others.  

Regulated industries are struggling to balance these concerns with the speed and lower cost with which they are able to generate insights using analytics in the cloud.

The latest controversy comes just a week after the NHS handed Atos a contract to build the controversial care.data patient information database.

Atos has drawn criticism for its handling of a separate contract to carry out health checks on people claiming benefits. 

What do you think? Should PA be congratulated or chastised? Have your say below...