ME Bank opts for automation to simplify DR testing

An overhaul of virtual environment testing processes has helped operations teams at ME Bank speed up annual checks of the bank's virtualised IT infrastructure through automation.

Disaster recovery testing is critical to ensure the business continuity capabilities of a completely virtualised application environment that the bank introduced during a four-year, $57 million overhaul of its core banking systems.

The operations team had previously run the testing using a complex process that infrastructure services manager Matt Forder says took around three months to execute from start to finish.

The immense effort, time and money involved – ME Bank technicians effectively had to create, test and decommission a parallel virtual environment containing a copy of the entire banking environment – meant this testing process could only be completed once per year.

“This process was mainly done with engineers,” Forder said. “It cost a lot of money because it involved a whole lot of manual effort.”

A process review – predicated on three key tenets including risk reduction, introduction of automation, and simplification of the recovery process – identified potential improvements like implementing Zerto Virtual Replication, a specialised hypervisor-level toolset that streamlines the replication and recovery of data between virtual server environments and cloud platforms.

Even casual consideration of the benefits of an automated solution promised enough cost savings to justify the business case for introducing the Zerto technology, Forder said.

"We were able to show that we could offset the investments we were making into this by the savings we made from not having to run a whole bunch of redundant infrastructure.”

As the team began testing the new approach it quickly became clear that the automation technology would dramatically streamline the testing process.

For example, the testing team had previously had to set up a separate IP address space for the bank's disaster recovery data centre, then reconfigure applications or “do a lot of DNS magic to mask the different IP addresses", Forder said.

In the new environment, the team was able to use virtual machine cloning to manage a single logical IP address space across both data centres – “which really helped reduce all the complexity in the recovery process”.

Instead of having to duplicate and track the creation of the copy of the virtual environment and its testing, the bank has now automated the DR testing process and dramatically shortened tasks such as spinning up virtual machines – which now takes 60 to 90 minutes less per virtual machine.

“During the recovery process, that's valuable time,” Forder said.

Better management of the virtual infrastructure meant “we were able to decommission a whole bunch of virtual infrastructure that we didn't require anymore. This generated fairly significant operational cost savings” for the organisation, he said.

The same testing process that used to take 3 months can now be completed in just 3 weeks, allowing the team to run the tests more than once a year.

“We are in a position now where from the set applications deployed on virtualised infrastructure – and in our case pretty much everything is – we are able to test them fairly smoothly,” Forder said.

“In the past we typically onboarded applications into the new environments, but now we just tick a few boxes, add those assets into a configuration, and walk away.”

Because the Zerto environment spans operating system and application environments – the bank's SQL Server databases are still replicated at database level, although that may change – adding disaster recovery capabilities to ME Bank's existing and evolving applications has become far simpler than in the past.

“That has significant cost savings for projects because teams don't have to worry about a solution design for disaster recovery,” Forder said.

“We now have a platform that eliminates data loss and downtime, and gets you back up and running quickly. Recovery point objectives are no longer a day, but are measured in seconds. That just comes out of the box and it's a huge benefit for our projects.”

The automation overhaul is still ongoing, with around half of processes automated and engineers now focused on automating the start-up of applications during the DR testing process.