NAB is modernising data pipelines for its Ada platform

NAB is going back and standardising the way data pipelines for its Ada data platform are built, which will lower run costs and make new recovery features available.

NAB's Dheeraj Puli.

Head of data reliability engineering Dheeraj Puli told the Databricks Data+AI Summit in San Francisco that the bank is standardising on Spark Declarative Pipelines (SDP) across more than 1600 data pipelines.

Ada is considered to be “chapter two” of the bank’s data journey, first breaking cover back in 2022.

It has Databricks on AWS at its core, and replaced several large data platforms, including the NAB Data Hub (NDH) and a 26-year-old Teradata environment.

However, there are already opportunities to modernise it.

As is common among Databricks customers, NAB has its data in the platform organised or structured according to a so-called medallion architecture.

Raw, unaltered data comes in at the bronze layer; it’s then cleaned up in the silver layer ready for use by AI/ML models and the like or curated for more specific business uses at the gold layer.

Data typically moves through a multi-hop pipeline as it progresses through the medallion system.

Ingestion of raw data to the bronze layer is already standardised, Puli said.

“When we started, bronze was actually using SDP from day one,” he said.

The opportunity is to modernise the data pipelines used to transform data for the silver and gold layers.

These pipelines are still based on Spark but were built in a variety of ways - and it’s this customisation and complexity that the bank wants to now take out of Ada’s operation.

SDP represents a different way of building pipelines. Notably, the build is “declarative”, meaning that rather than coding every extract, transform or load (ETL) step, engineers “define what the data should look like” and then Databricks - or more specifically, its Lakeflow platform - handles the required “orchestration, state management, and incremental processing” in the background to make it all happen.

Work is already well advanced at the silver layer, with new pipelines being built using SDP and older ones being modernised.

“All the new silver [pipelines] are going on SDP,” Puli said.

“We completed a majority of the uplift of the existing custom Spark SQL [pipelines] to SDP.

“Now, we are in the middle of testing those jobs for the safe cutover between custom Spark to the SDP.”

A case study published by Databricks to coincide with the US presentation suggests that work at the silver layer is about 50 percent done; at the event itself, Puli said that “currently we have 3800 SDP pipelines running in silver”.

The next step will be extending the use of SDP to the gold layer data as well.

“The long-term goal is a fully declarative, end-to-end streaming architecture-from bronze to gold,” the case study states.

“Looking ahead, NAB aims to become the first bank to run 100 percent of its pipelines on SDP, setting a new standard for how large financial institutions build, operate, and scale data platforms.”

Aside from consistency and efficiency in the way pipelines are developed and run, the bank is anticipating savings from its adoption of SDP.

“We expect 15 percent lower run cost when we turn off all the [custom] Spark jobs and then run [pipelines] only on SDP,” Puli said.

“Currently we run both in parallel - our SDP and then custom jobs.”

Moving to SDP has also enabled NAB to privately preview new features and to run proof-of-concept testing.

One of these features, “rewind and replay”, allows users to recover a pipeline that has failed to a previously consistent state.

Puli said the bank elected to join the private preview owing to the emergence of a finance-specific use case.

“A few months ago, we had a business requirement [with] one of the finance systems [where] they don't want to load data unless it's fully reconciled,” he said.

“For any reason, if this data is not reconciled, they want us to [be able to] roll back to the previous state where the data is reconciled.”

Without a ready solution at the time, Puli said that NAB “built a tiny module [of] custom code to meet our timelines and the business requirements.”

However, it appears the “rewind and replay” feature could provide a native way of meeting the specific rollback business requirement.

Ry Crozier attended the Databricks Data+AI Summit in San Francisco as a guest of Databricks.