NAB's cloud devs get free from locked-down IT

NAB developers are making progress at finding ways to build and deploy code to the cloud despite a heavily locked-down IT environment and limited allowances to run tools on local machines.

NAB's Lupco Trpeski.

Speaking on the sidelines of the AWS Summit in Sydney, senior automation and DevOps consultant Lupco Trpeski indicated NAB had come a long way on developer experience in the 18 months he had been at the bank.

Much has been said about NAB’s shift to public cloud - from its Cloud Guild that has trained 4200 people in a year, to its goals to re-home 35 percent of IT applications in the cloud within three to five years.

But the ‘people’ side of the project has only been recently laid bare.

At AWS’ US-based summit at the end of last year, NAB revealed it was using a lot of AWS technical staff to perform migration work after a decade of IT outsourcing left it with few skilled internal resources of its own.

The bank’s appearance at the Sydney version of AWS Summit this week shows those that did join NAB’s internal workforce ran into problems trying to work within the boundaries of the bank’s secure environment.

“Day one [at NAB] for me 18 months ago was really complex,” Trpeski recalled.

“We couldn't deploy anything. It was really hard to actually get anything installed on your laptop - and we only had Windows laptops - so there was no tooling.

“Getting access to [AWS] accounts was complex, and then when you were in the accounts, it was actually really hard to deploy services.”

Developers were used to the freedom of being able to choose their tooling and being able to work on Mac or PC. However, upon arriving at NAB they quickly found the environment was Windows-only and that elevated privileges - including administrator rights over the laptop they used - were not an option.

“We really struggled when we were on Windows,” Trpeski said.

“[NAB] gave you Windows and Word, and you could write documents and check your emails, but you couldn't do anything else on there. I don't want to talk specifically about that.

“But essentially we found that the only thing that we could do was install things that didn't require us to have local admin access.

“We found that we could use Cygwin or Babun and install those without local admin access and that would allow us to be able to build tooling like the AWS CLI [command line interface] on our laptops and have like an emulated Linux environment.”

Developers now also use an integrated development environment (IDE) hosted within AWS. The bank is an adopter of AWS’ Cloud9, which allows developers to write, run and debug code in a browser.

“If you're in America or Europe you can run [Cloud9] up as an AWS service but if you're in Australia, you have to build it yourself, so it's a web-based IDE run on EC2,” Trpeski said.

“Running Cloud9 as an IDE in our AWS environment allowed us to do more than what we could do on our laptops.”

Trpeski said that Cloud9 had been a “game changer” for the bank’s developers, but it was one component of a broader number of changes aimed at making developers’ lives easier.

The bank has recently relaxed its Windows-only rule and, through a project called Mac@NAB, now allows developers to use Apple machines.

That saw some additional freedoms trialled and introduced, including the ability to run Docker locally on developers’ laptops.

“It was the first time we were able to do that, and that only came in about three months ago,” Trpeski said.

However, Trpeski noted that “just being able to install things locally wasn't enough for us.”

Continuing the locked-down theme, access to NAB’s AWS accounts and environment is also challenging.

“In a traditional organisation, you would get an AWS account, it would have internet gateways, and you would have private and public subnet and NAT gateways, and if you needed something, you could just go and grab it from the internet,” Trpeski said.

“We don't have those. We only have private infrastructure. All of the connectivity to our accounts is via an AWS DirectConnect, and the only ports that we can actually access in our application VPCs [virtual private clouds] are Port 80 and Port 443, so even if you wanted to SSH [secure shell] into an instance in our production application environments, you couldn't.

“It is so secure because it’s what people expect of us [as a bank], but it can also make things really challenging for a DevOps person.”

Trpeski continued: “Because we don't have internet connectivity in our accounts, we have to access all the services through VPC endpoints." A VPC endpoint enables users to privately connect a VPC to supported AWS services.

“So to actually get files into an S3 bucket or to be able to hit our API gateways, we need to actually be within the AWS environment,” he said.

NAB has partially resolved this by using Cloud9 not only as an IDE but also as a bastion host; essentially, as a primary access point from which to access other parts of the AWS environment.

“We used to have bastions in shared accounts, and we had a dedicated bastion that the security team would set up for us, we could get into that bastion, and then that bastion would allow us to get into our production VPCs,” Trpeski said.

“With Cloud9, we found we could actually still maintain that bastion role, but have something that did a better way of doing it.

“So you can use Cloud9 as a bastion but you can also use it as a fully functional IDE.”

Augmenting Cloud9

In addition to Cloud9, Trpeski said other projects had also made developers’ lives simpler.

“We had a continuous compliance project where the security team can use Lambda functions within our accounts to actually monitor IAM [identity and access management] roles that we create, and if anything in those IAM roles doesn't meet our compliance standards, it dynamically changes the IAM role so that it does comply with our standards,” he said.

The upshot of this for developers was that they could “create [their] own IAM roles for the first time.”

Trpeski said that another product is being developed in-house at NAB to enable “a fully autonomous production pipeline, all the way from devtest into our production accounts”.

This was alluded to at the US summit, where NAB said it had set up "guardrails" that remained invisible to application teams unless a particular limit was breached.

“It has stage gates where we can run it through security controls, we can run it through compliance, and we can run it through software assurance,” he said.

“These are the things that we didn't have 12 months ago, and we could only really do them because we partnered with AWS to create these products for NAB to be able to launch services safely and securely within our environments.”