Boral cuts US$1m a year from its cloud costs

Boral, a listed maker of building products and construction materials, has revealed it was able to cut US$1 million (A$1.35 million) a year from its cloud costs through a concerted effort to find savings.

Boral file image (Credit: Boral)

The company, which is perhaps best known for its ubiquitous concrete mixers, has slowly made public a large-scale cloud migration over the course of the year via a series of videos and webinars.

In the most recent video, published by AWS in late October, Boral’s cloud services product owner David Marks said the company had exited its data centre as part of the migration.

“We’ve been really successful in cloud at reducing our costs and increasing our efficiency,” Marks said.

“We’ve successfully shut down 50 percent of our [cloud instance] fleet most of the week to save costs, [and] we’ve ‘rightsized’ most of our resources so we’re just paying for what we need. 

“We’ve got more work to do in this area and our next target is backups, where we think we can make some significant savings and increase the resilience of our environment.”

In a separate video, from an August webinar run by Versent, Marks describes in detail the company’s substantial efforts to cut the costs of its cloud-based environment with the help of Versent.

Versent is an AWS-focused cloud consultancy involved in many blue-chip migration and digital transformation projects.

Marks used the Versent webinar to reveal the topline cost savings that Boral had achieved, as well as a breakdown of where the largest savings had been realised.

“In our case we’ve saved $1 million US per year, just by checking [the environment] regularly, by emptying the sandbox, by adopting savings plans, by removing provisioned IOPS, and by configuring auto shutdown for RDS and EC2,” Marks said.

Auto shutdown

Enabling auto shutdown led to over half of the annual savings, with EC2 auto shutdown accounting for 45 percent and RDS [Relational Database Service] auto shutdown a further 10 percent of the US$1 million a year being saved.

“Terminating an instance or resource generates the biggest saving,” Marks said.

“If you don’t need something, get rid of it. The cost is then gone forever.”

Marks said that Boral essentially gamified auto shutdown of unused or under-utilised instances by broadcasting regular emails that ranked product teams based on how much each had saved.

“This idea came out of an AWS cost optimisation immersion day that I attended,” Marks said.

“It’s a very simple idea: broadcast an email to a lot of people with a table show who was the best, then sit back and watch people try to get to the top. 

“After I started sending this series of emails, people kept calling: ‘David, how can I get to the top?’ ‘I shut down all of these systems, how come I’m not further up?’ 

“It just created a lot of competition, and by doing that over a period of time, the adoption of auto shutdown increased significantly.”

Marks said that Boral saved US$400,000 a year on EC2 costs alone.

“On August 11 when I took a snapshot, we had 558 EC2 instances,” he said.

“If all those instances were running 24x7 we would consume 93,000 hours of compute time each week. 

“By applying auto shutdown across our fleet we are saving 43,000 compute hours a week. That’s 43,000 hours of compute time we don’t have to pay for. 

“With a similar fleet to ours with 500-600 servers, you could easily save US$400,000 per year, simply by shutting [instances] down.”

On the RDS side, Marks said Boral started to look at cost optimisation by writing a script to “analyse each RDS instance and report back … the number of connectiosn per instance per hour”.

Almost immediately, the company was able to identify databases running 24x7 with “no connections” that were running up costs.

Armed with the evidence, Marks challenged the business to run RDS instances for less time.
“Instead of running 24x7, why don’t we run it 12x5 [12 hours a day, five days a week],” he proposed.

“After they accepted that and we implemented it, each week these RDS instances were running 60 hours rather than 168 hours per week.”

On two test RDS instances, Marks said Boral’s running costs fell from US$1600 a month down to US$600 a month - and he said further savings were made by consolidating both databases to run on a single RDS instance.

“Those same two RDS databases were used in a single business unit, and when we saw that the connections weren’t that high, we recommended they put two databases on a single RDS instance,” Marks said.

“They agreed to that … so that then reduced the cost by 50 percent of what it was at that point in time. 

“If you can do that for multiple databases across your fleet, it’s just more and more opportunity for cost savings.”

Provisioned IOPS

The second largest chunk of savings came from reducing - and ultimately removing - the use of provisioned IOPS solid state disk (SSD) backed storage.

Provisioned IOPS volumes “are the highest performance elastic block store (EBS) storage volumes designed for critical, IOPS-intensive and throughput-intensive workloads that require low latency”, according to AWS documentation.

“What they [AWS] don’t tell you in the brochure is that when you use migration tools, they look at the source system and make some guesses about the amount of provisioned IOPS and other resource allocations that you need,” Marks said.

“When we received assistance to migrate 100s of instances, the costs were high, and when we started to analyse we found that provisioned IOPS was very expensive.”

Marks said he assigned one of his team of seven that looks after Boral’s infrastructure “to write a script that showed us firstly the amount provisioned IOPS that were allocated and also the amount of provision IOPS that we were actually consuming.” 

“It turns out that we were rarely consuming the amount of provision IOPS that were configured, and we were able to completely eliminate provisioned IOPS from our configurations,” he said.

That reduced the company’s cloud costs but not the performance of its environment, according to Marks.

“We were able to do that in a bulk lot for non-production instances and then we did that more slowly in December [2019] for production where people wanted to be more confident and we would test a bit,” he said.

“Provisioned IOPS is a bit like the Ferrari going to the milk bar. We didn’t need it. 

“With a similar environment [to Boral’s], removing provisioned IOPS could save up to US$330,000 per annum. That’s a lot, and is a very good tip if you are using any migration tools to migrate to AWS.”

Finding other savings

Outside of auto shutdown and provisioned IOPS, Marks detailed several other smaller actions that had contributed to the topline savings number.

He said Boral had learned early to “address spikes” in usage very quickly, and that it engaged “Versent managed services who provide 24x7 monitoring and alerting and also help us with improving the environment so it becomes more automated.”

He cited one example where a product team started using a new tool that, if left unchecked until the end of the billing cycle, could have cost Boral upwards of US$30,000 in excess charges.

Marks said further savings were being made by regularly emptying the company’s “sandpit” environment where teams can test services and configurations pre-production.

“People create resources and forget to delete them,” Marks said.

“Creating a script to clean out a sandpit regularly is a great way to do it. It runs every Monday and terminates any resources in sandpit that are older than 30 days. 

“The people that use the AWS environment at Boral know that that’s the case, and if they forget to do their own cleanup, we clean up for them. 

“This could be saving US$10,000 to $20,000 per year for people that forget to terminate resources.”

Additionally, Marks said Boral had enforced a metadata tagging standard for AWS resources “from a fairly early stage of the journey”.

Tagging is used to categorise resources as well as to establish ownership.

“All the automation that you will consider will be dependent on a solid tagging strategy,” Marks said.

“You’ll be able to make informed decisions when you know what your environment looks like, and you can easily make decisions that make sense. You’ll have a clear picture of your fleet.

“AWS has a hard limit of 50 tags for any resource and we’re using probably about 42. 

“I think it’s also very important to keep checking that the data in your tags is accurate and up to date. You’re going to struggle to do it by yourself so set up automation where you can to update tag values, and identify champions within your organisation to help you maintain those tags.”

Savings plan

With a better handle on its cloud environment and needs, Boral entered a savings plan with AWS, which commits to a certain amount of usage in exchange for lower pricing.

“Once you have all your rightsizing done, once you’ve terminated instances, once you know which instances are going to run 24x7 and which ones are going to run for shorter periods of time, that’s the time to consider reserved instances and a savings plan,” Marks said. 

“AWS gives you a discount for committing to a period of time, and that minimum is 12 months but it can be two or three years. 

“The amount of the discount varies depending on the commitment term, the instance and OS types, and a number of other factors, and it works best if you are running those instances 24x7.”

Marks said Boral had achieved a “16 percent reduction in eligible costs” from its savings plan.