In 2014, HSBC’s data assets sat at 54 petabytes. Slightly more than two years later, that figure has doubled to more than 100PB.
Needless to say, the finance giant had an infrastructure challenge on its hands.
Two years ago the bank - which has historically either built or customised technology - decided to switch to a lease-over-buy mindset.
Late last year it revealed the first result of that change: a global replacement of its ERP and EPM systems with Oracle's cloud-based Fusion platform.
Now the bank is turning its attention to its IT infrastructure.
The record levels of data the bank is dealing with is the result of customers adopting digital channels more aggressively, according to HSBC global CIO Darryl West. The trend has forced a change in thinking about how information is handled.
“Our journey in data is very similar to other companies,” West told the Google Next Cloud conference in San Francisco today.
“We have lots of good old fashioned databases, all our core, product systems that have been around for 30 or even 40 years. They are robust and do a great job, but they don’t have the database structures that allow us to do the data analytics and machine learning we need to do.”
The bank has historically extracted data from its core systems into “traditional” data warehousing platforms, which “worked well for many years, but in recent times have become expensive to run and difficult to use”.
So three years ago HSBC took the plunge and embedded itself into the then-evolving open source Hadoop ecosystem.
It hasn’t been an easy road, West said.
“All of this has been done on-premise, we’ve had to provision [lots of] physical infrastructure, build out data centres, and hire new talented people who have an understanding of these new technologies,’ he said.
“So we thought about this very carefully in the last year and said: ‘at the core of our business, we’re a bank, but also we have a significant technology company embedded within the organisation’.
“And the question I was asking the management team is 'do we really want to compete against the cloud providers, are we really going to try and do what they do as well as they do it?'”
The answer was no, West said.
“It was better for our business to adopt a cloud-first strategy."
HSBC started working with Google on its cloud platform product about six months ago.
Five initial use cases - chosen because they involved large datasets and needed intense computing capability in short bursts - were first off the rank: anti-money laundering, finance liquidity reporting, risk analytics, risk reporting, and valuation services.
For its anti-money laundering obligations, HSBC has a set of applications that monitor billions of customer transactions to identify patterns and extract what could be nefarious activity.
“This is an application that clearly requires a massive dataset, great computing capability, but also machine learning to identify patterns," West said, making it a prime option for the cloud platform.
The four other use cases similarly involved “billions” of transactions and significant compute demands.
“We were faced with the question of whether to build out a new data centre and put thousands of servers out there to do this activity, or should we work with someone who does this as their core business?” West said.
While it’s only “early in the relationship” with Google - HSBC is still in pilot mode but hopes to get DataFlow and BigQuery into production within its environment soon - West indicated he expects continuing growth in data assets means HSBC’s days of owning and operating infrastructure are over.
“Massive datasets need to be managed. We also know we’re going to be pushed to be faster. Our customer base is dragging us into a new world, by consuming services in digital channels that require us to be agile and fast to market,” he said.
“It’s going to be bigger and more complex, and we’re going to need to partner with Google and others to use the best technology to solve these problems in our business.”
The bank is currently working with Google on a construct that will meet all its risk, compliance, privacy, and security obligations as a heavily regulated financial services provider, whilst allowing it to be “agile” in adapting to rapidly changing customer demands, he said.
Allie Coyne travelled to Google Cloud Next as a guest of Google
