NAB approved to use serverless in data environment

NAB has revealed it is starting to make use of serverless compute as part of its data ecosystem, with an 18-month lead time required just to gain regulatory and internal approvals.

Speaking at Databricks’ Data+AI Summit earlier this month, distinguished engineer for data platforms Daniel Antoinette said that serverless was seen as a way to ensure data users could access “fast, safe, reliable and scalable” compute resources to underpin their analysis or AI work.

It avoids management overhead associated with resources that would otherwise be spun up in NAB’s AWS accounts.

But as Databricks allocates and manages the compute, and the use cases are all data-related, a number of approvals - internal and external - were required by the bank to go down this path.

“You might be thinking: Surely we can't use serverless in a highly regulated company, where the data leaves our control, but we actually can,” Antoinette said.

“While we've been able to get approval to be able to use serverless … It was a very time-consuming process. 

“There were a number of different risk assessments that we needed to do - so third-party risk assessment, customer risk assessments, the list goes on, and we also had to consult with our regulator just to be able to get the approval to use serverless. 

“That took about six months to do, which is quite an amount of time.”

NAB is no stranger to challenging regulatory approval processes when it comes to cloud use, previously needing the Australian Prudential Regulation Authority’s (APRA’s) blessing to scale its workload migration into public cloud.

Antoinette said that the regulatory approval timeframe for Databricks serverless had some “upside” for the bank, in that the vendor was able to augment its serverless offerings in that time.

“The benefit of it taking that amount of time was that some of the functionality from Databricks came out that we needed to be able to properly implement serverless and adopt it,” Antoinette said.

“Stuff like internet on/off, secure egress policies or seg[regation] policies, [and] the budget policies so that we can monitor and control costs - all of those things were required for us to be able to properly adopt serverless but also be able to provide the service to our end users as well.”

Regulatory approval timeframes, however, paled in comparison to the internal effort needed to enable connectivity between serverless resources “and other NAB-managed systems” securely.

NAB is using AWS PrivateLink to enable that connectivity but is still in the process of optimising how this works in practice.

Antoinette noted that “there [are] a lot of concerns with enabling this sort of connectivity within a bank.”

“Let me tell you: if you thought that it took a long time to get approval to use serverless, we actually experienced the first birthday with the back-and-forth conversations to be able to get this enabled within the bank,” he said.

“That was just the conversations - not even the controls or anything like that, as well as the implementation side of things. 

“And so, with a lot of collaboration - with Databricks, with our security, network and cloud architects - we were able to come up with a minimum set of controls that needed to be enforced to be able to minimise the risk to enable this connectivity.

“We were also able to come up with an architecture that allowed us to meet those controls, so that we could implement something like this within the organisation.

“There are a number of different hops - different gateways, firewalls, balancers - that we need to go through just to be able to implement those controls to establish that connectivity.”

In a slide deck, NAB said the multiple hops minimised the risk of incorrect or unauthorised access. The setup also allowed the bank to "control exactly what repositories and paths Databricks has access to."

Antoinette noted the construct “isn’t great in terms of latency and costs” but has served its purpose - “allowing us to tap into the full potential of using serverless, while in parallel we iterate and can evolve and understand how we … can implement it better to minimise the amount of hops and minimise the cost associated with it.”