ANZ uses AI to move 1.7PB data platform to AWS

By

Retires legacy Hadoop environment in one-third of the expected time.

Key points

  • ANZ migrated its enterprise big data platform from on-premises Cloudera Hadoop to Amazon EMR in six months, a year faster than the 18 months anticipated.
  • The move covered 1.7 petabytes of data across 300 million files, 142,000 tables, 4500 automated workflows and 4000 views, with agentic AI cutting migration effort by an estimated 35 percent and halving defects and rework.
  • Almost 80 terabytes has been tokenised to protect sensitive customer data, running about seven times faster and cutting tokenisation operating costs from over US$10,000 a day to under US$3000 a month.
ANZ uses AI to move 1.7PB data platform to AWS

ANZ Banking Group has moved one of its largest data platforms off an on-premises Cloudera Hadoop environment onto Amazon Web Services in six months, crediting AI-assisted engineering with cutting a year from the schedule.

The bank said its enterprise big data platform now runs on Amazon EMR, AWS's managed service for open source data processing frameworks such as Hadoop and Spark.

Formerly called Elastic MapReduce, EMR runs those frameworks on rented clusters of virtual servers, which saves customers from building and maintaining the hardware themselves.

The migration covered a very large 1.7 petabytes of data volume across 300 million files, 142,000 tables, 4500 automated workflows and 4000 views.

ANZ told iTnews the work largely followed a lift-and-shift approach, in which workloads are moved to the cloud with minimal redesign.

The bank had anticipated the job would take 18 months.

Asked what its comparisons were measured against, ANZ pointed to the scale and complexity of the estate, including legacy technologies and complex access controls.

Chief data and AI officer Kai Yang told an AWS media briefing that the bank set up a "hothouse environment" that put its business and technology staff in one room with AWS engineers.

Yang said the team used coding agents, and built troubleshooting agents to reduce defects.

ANZ said agentic AI reduced migration effort by an estimated 35 percent and cut defects and rework by about half compared to traditional approaches.

Engineers used a troubleshooting assistant built on Amazon Bedrock to find and fix performance bottlenecks, although ANZ did not name the models involved beyond saying they were internally approved.

The project showed AI-assisted engineering could "accelerate complex delivery while maintaining a disciplined focus on security, resilience and quality", group chief information officer Donald Patra said.

ANZ's platform supports the bank's payments, customer remediation, and business and private banking operations.

Sensitive customer data had to be protected before it could be moved to the new environment, which ANZ did through tokenisation, replacing sensitive values with stand-in tokens.

Almost 80 terabytes has been tokenised so far.

ANZ said that the capability ran about seven times faster than an initial approach, and cut operating costs from more than US$10,000 a day to less than US$3000 a month.

The bank confirmed those cost figures cover tokenisation only, not the platform as a whole, and did not describe the earlier approach.

AWS Professional Services and unnamed AWS partners worked on the migration alongside the bank's own engineers.

ANZ chief executive Nuno Matos said last year that the bank would depend less on third parties, and use them only in special cases such as to speed up delivery.

Yang said the modernisation was needed to give the bank good quality data and a scalable lakehouse architecture, a design that combines data lake storage with warehouse-style management.

He said ANZ was also building a reusable AI platform that would run on-premises as well as on cloud services such as AWS.

Yang added that the enterprise AI control plane ANZ revealed at its annual results last week would provide continuous monitoring, auditability and human accountability for the bank's AI systems.

Add iTnews as your trusted source

Got a news tip for our journalists? Share it with us anonymously here.
Copyright © iTnews.com.au . All rights reserved.
Tags:

Most Read Articles

ANZ stands up an 'enterprise AI control plane'

ANZ stands up an 'enterprise AI control plane'

NAB deploys AI for staff called 'Ruby' that is built on Anthropic

NAB deploys AI for staff called 'Ruby' that is built on Anthropic

Westpac reveals AI cut data warehouse migration cost from $70m to $35m

Westpac reveals AI cut data warehouse migration cost from $70m to $35m

Australian Unity taps former Optus CIO as tech leader

Australian Unity taps former Optus CIO as tech leader

Log In

  |  Forgot your password?