ANZ Bank goes all-flash for storage

ANZ Bank has lifted the lid on a three-year effort to move away from traditional spinning disk to an all-flash storage environment.

The project, first revealed in late 2015 by an ANZ executive, has largely remained out of public view, despite being one of the largest all-flash environments in Australia and the first of its kind for a bank in Asia.

Associate storage engineer Patrick Hall told IBM’s Think 2018 conference in March that the bank had traditionally been an EMC VMAX storage area network (SAN) shop.

However, in 2015 the storage team - which consists of about four full-time staff - found itself reporting to new management, and that provided the impetus for a change in the storage environment.

In short, the decision was taken to pursue all-flash.

In 2015, all-flash was still considered by some to be a risky proposition, although adoption increased throughout 2016 and into 2017, particularly as spinning disk and flash approached price parity.

ANZ decided to move away from being a single vendor environment, and instead “signed deals with HPE, IBM and Pure Storage to supply us with the all-flash technology”, Hall said.

The bank also bought IBM SAN volume controllers (SVCs) to provide virtualisation across the multi-vendor arrays and generally to manage the new environment.

“For the last three years we have been cutting over completely to flash disk and moving away from the traditional spinning disk,” Hall said.

“Today at ANZ we are in what I would say is a tuning process.”

A side benefit is that the architecture - as well as the flexibility and performance characteristics of the all-flash environment - helped the storage team meet its obligations under ANZ’s bank-wide New Ways of Working program, which is the umbrella name for the institution's agile push.

For storage, agile means being able "to deliver quickly and accurately at the drop of a hat as well as ensure the capacity and performance is available for anyone to use at any one time,” Hall said.

First phase

The first phase of the all-flash migration used flash arrays from Pure Storage and HPE’s 3PAR.

Hall sized that part of the environment only by saying the bank had bought “a lot of Pure Storage arrays”.

The bank had several reasons for wanting to put the 3PAR and Pure Storage arrays behind an SVC.

“The decision was made to connect the HPE 3PARs and the Pure Storage arrays via Fibre Channel to the IBM SVC,” Hall said.

“This was done for two reasons. One, Pure Storage didn’t have synchronous replication at the time, and this is something we definitely required to adhere to various regulatory requirements; and two, HPE 3PAR would also go to the SVC because the user interface of the SVC is much simpler than the 3PAR.”

The bank - and users - had grown accustomed to synchronous replication from the EMC era. It offers instant failover for transactional applications in the event of a storage issue.

“We were used to the technology and we were used to the fact that when we specified synchronous replication it just worked,” Hall said.

“For our customers - or the people that build the applications or require virtual servers - they too came to expect the same level of confidence that synchronous replication brought - that it just worked.

“But when we procured the Pure Storage arrays that sit behind the SVCs as backend storage, at the time we procured these they did not provide an adequate synchronous replication solution for us.”

Hall said that SVC's ability to handle synchronous replication was “an obvious win” for IBM, given the technology had been sold into ANZ for some of its other storage management benefits.

“An extension to this is it also handled the synchronous replication efforts for the 3PARs,” Hall said.

“Despite the fact they have their own [synchronous replication technology], the decision was [taken] that in order to simplify the management of the environment everything will go behind the SVC.”

Enter IBM flash storage

For eight months, the bank ran with the SVC virtualising both HPE and Pure Storage flash arrays but utilisation issues developed.

“During this time we ran fairly close to the wire in terms of utilisation, hitting almost 100 percent at some points and going over 90 percent the majority of the time,” Hall said.

Utilisation hadn't been an issue with the bank's disk-based environment, mostly due to its size.

“We were coming off some pretty significant VMAXs," Hall said.

Hall said the IBM sales team on the ANZ account mobilised in response.

"They did some pretty significant selling as they were able to convince ANZ management to lend us some FlashSystem 900 (FS900) arrays for testing," Hall said.

“What that effectively did was IBM got their flash storage into ANZ and it became another option to add to the mix alongside the 3PARs and Pure Storage arrays.”

The FS900s allowed the bank to alleviate some of its capacity concerns within the new storage environment.

“[They] effectively opened the door for IBM to be able to offer us some new options,” Hall said.

“The FS900s worked great [as] small units to provide a decent capacity pool directly to the SVC.

“However, being that we had become used to large capacity arrays we continued to ever so politely harass our ANZ-to-IBM relationship manager with the fact that we wanted a big array because managing a lot of smaller arrays [in] a pool can be cumbersome and slow.

“The bank listened to us and decided on four [Storwize] V7000s and a total capacity of roughly 400TB per unit.

“This in itself was pretty significant to put us in a space where our overall utilisation across our environment dropped away from the danger zone and meant we could continue to migrate applications and servers to being behind the SVC.

“It basically gave us some breathing room.”

Hall said that IBM’s flash hardware - the FS900s and V7000s - “has been very reliable and easy to use.”

“When they are behind the SVC they simply hand off to the SVC the role of compression and deduplication,” he said.

“The arrays do what they need to do which is provide highly available, good performing flash storage.”

Some challenges, but promise largely realised

The bank also started using two additional IBM technologies: HyperSwap for high-availability active-active access to storage volumes, and Spectrum for storage monitoring and analysis.

“Previously at ANZ we used to use [EMC’s] ViPR [for storage analysis] but that came with a big learning curve and was challenging to use,” Hall said.

However, Hall noted that even Spectrum had “had a fairly rough run at ANZ” so far.

“I don’t know if it’s the size of our environment or the sheer fact that no matter what storage management analysis tool we install it never seems to perform as it should,” he said.

The bank had also run into challenges with the SVC technology, though Hall noted this was again likely to be the result of ANZ being used to working with huge arrays of spinning disk.

“We’ve hit just about every limitation I think you can possibly hit on the SVC - from the number of volumes, the number of MDisks, the number of consistency groups, the memory for replication, the I/O group and node performance limits just to name a few,” he said.

“To qualify this, a lot of it was a result of us.”

Overall, Hall said that all-flash had resulted in performance improvements for the bank’s applications.

He said ANZ is also continuing to pursue technical enhancements to improve the way flash arrays - particularly Pure Storage’s - interact with IBM’s SVC technology.

“I’m not saying Pure Storage and IBM aren’t compatible - a lot of time they work well together,” Hall said.

“However, you need to think of these two technologies as highly competitive colleagues, always trying to outdo one another and yet constantly forced to work together.

“If you take Pure Storage, it is the polar opposite of the FS900 and V7000s. By default, compression and dedupe is turned on which means you have the SVC wanting to take control of those things but can’t because the Pure Storage array is doing it.

“In addition the code upgrade process of the Pure Storage arrays almost always throws errors and alerts into the SVC.

“But hope is not lost. Given the fact ANZ do have a lot of Pure Storage arrays, and we’ve adopted the SVC for virtualisation across our entire environment, we have been able to put in numerous enhancement requests [for the technologies to work better together].”

Ry Crozier attended IBM’s Think 2018 conference in Las Vegas as a guest of IBM.