How cloud exposed EB Games' real dev challenge

When EB Games moved e-commerce off its own infrastructure and into the cloud, it removed key roadblocks to the company’s omnichannel ambitions.

But it also eliminated the possibility of technology being the root cause of other bottlenecks - and that realisation led to important shifts in developer culture.

Senior technology manager Kevin Clarke told Adapt’s Cloud & DC Edge 2018 conference - co-organised by iTnews - that the retailer had been better able to support significant sales events after shifting e-commerce operations to run on AWS platform-as-a-service.

“Omnichannel is the fastest growing part of the Australian business so we wanted to make sure that we were moving towards an environment that supported the exponential, burstable growth we’re seeing in that space,” Clarke said.

“We wanted to enhance the customer experience around our peak load days. Traditionally when we had peak load the user experience would suffer.”

As Australian consumers embrace sales events like ‘Black Friday’, companies like EB Games are opening themselves up to potentially enormous temporary traffic spikes.

On Black Friday last year, EB Games’ click-and-collect sales set new company records, and exceeded even the company’s own best-case expectations.

To put some of these spikes into perspective, the company saw a greater sales volume through its website in one 24-hour period in 2017 than it did for the whole of 2010 - the year when it first started out on its e-commerce journey.

“The scale has really changed the dynamics,” Clarke said.

“If we hadn’t moved into the cloud I don’t think we could have supported that kind of growth.”

EB Games’ move to cloud was also designed to improve the way the e-commerce platform was enhanced and new features were added.

It allowed the company to properly pursue continuous integration and continuous delivery (CI/CD), which it saw as a way to improve time-to-market for new features and functions.

“We saw that our traditional infrastructure stack was holding us back in terms of delivering software and trying to move faster,” Clarke said.

“We really hit a roadblock [to] continuous delivery being on-prem; going to the cloud was definitely seen as a tool we could use going into continuous delivery.”

One thing thought to be slowing development processes down was that all code passed through a single quality assurance (QA) environment.

The answer seemed simple enough: use the cloud to stand up more QA environments, split the work between them and take advantage of test automation to reduce manually-repetitive assurance work.

Clarke validated his theory.

"We did a lot of test automation - basically automating regression tests - and the idea was that we would have these running each night," he said.

"We found it was reducing the need to have manual testers doing repetitive tasks but it wasn’t increasing the quality [of our output]."

The result of that work forced a rethink of the entire problem.

“Before the cloud, it was easy to say ‘it’s all colliding in QA because we don’t have enough [test] environments’ and then to stop thinking about it,” Clarke said.

“Once you [discount] that, you’re left with some hard facts.”

For Clarke, it meant realising that continuous delivery was not being held back by a technical limitation but rather a cultural or behavioural one.

"I found that as we migrated and evolved into the cloud and brought the infrastructure, development and QA [functions] closer together into the whole development cycle, it's allowed us to analyse how we’re behaving and what we should be focusing on," he said.

"The key driver is quality. I could see that the problem was that we were not focusing on building a quality product and validating that earlier on in the development cycle.

“Moving to the cloud allowed us to identify our behaviour, but then changing it was definitely more challenging.”

Clarke looked to Toyota’s Jidoka quality control methodology for inspiration.

One of the car maker’s most well-known implementations of Jidoka is the (now-retired) Andon cord - a piece of rope that assembly workers could pull if they spotted a quality problem on the line.

“That’s assessing quality as you build it,” Clarke said.

“Compare that to if the car’s finished and you’re just making sure it doesn’t have any defects. If it has a defect you have to go back and fix it so you’ve just wasted all that [build] time.

“I think that analogy is very true to development. If you’ve got [development] silos it makes it very hard to see [the process] as a whole.”

Clarke needed the buy-in of his development team to redefine how they approached quality.

Rather than develop stuff and have QA test it - a model Clarke said gave developers “a kind of safety net” when it came to making sure their code didn’t have defects or “wasn’t going to break something” - he wanted to make developers more accountable for their output.

“We were already high-performing, the business was happy, there was no one saying you need to change what you’re doing,” Clarke said.

But Clarke saw more and more instances of CI/CD in web-based businesses, and he didn't want EB Games to be left behind.

"CI/CD's been happening for a fair while now and I think if you’re not doing it you’re going to be at a significant disadvantage," he said.

“For me, it was really challenging to say ‘I need to change the way we work’.

"People were unhappy because you’re really changing culture at a pretty core level, and you’re making people more accountable. That’s tough.

“But once we’d managed to focus on building a quality product, validating that earlier on in the development cycle and really getting a feedback loop working - which was a pretty big cultural change - our velocity increased a lot.”