How Sportsbet handled 12,500 Melbourne Cup bets per second

They call it the race that stops a nation – but for one chief information officer, Melbourne Cup Day was one of the busiest work days of the year.

Sportsbet chief information officer Simon Noonan.

At one point, during the 15 minutes leading up to the big race, Sportsbet was handling 12,500 transactions per second.

It’s a cracking pace even by the ultra-competitive standards of the sports betting market, where many consumers use multiple betting apps and where being late to market with new products can lead to a competitor knuckling over as the rest of the pack rides ahead.

For Sportsbet chief information officer Simon Noonan, the number one priority is to make sure the company’s core systems don’t collapse on the day of a major sporting event, while at the same time galloping forward in the race for new products and innovations.

Since taking on the role, Noonan has made an all-in bet on innovation, while making sure Sportsbet carries the correct weight when it comes to its internal IT infrastructure. 

Handling 12,500 transactions per second

According to Noonan, Sportsbet’s core OpenBet system was able to stay online and perform well despite the immense demand placed on it during this year’s Spring Racing Carnival.

“After this year’s Melbourne Cup, we now hold the world record for our core system in terms of transactions per minute – the first time this has been held outside of Europe," he said.

“Not only did we break records for OpenBet, we didn’t have any performance degradation or outage at all during the Spring Racing Carnival, right back to the AFL Grand Final, and a lot of that’s down to great teamwork, focus and discipline.”

The need to cost-effectively manage surges in demand, while simultaneously handling sensitive customer transactions, led the company to adopt a hybrid cloud environment.

“We use a mix of proprietary systems and vendors. We have a hybrid environment with a mix of on-premises data centres, SaaS [software-as-a-service] systems, private and public cloud,” Noonan said.

“We’ve have most of our data analytics in mix of private/public cloud, and we have a deep competency for that. Our core systems are hosted from our data centre in Darwin. Our core infrastructure is made up of one-third IBM kit, one-third Oracle and one-third Dell.”

This structure allows Sportsbet to move data-intensive app content, such as graphics and video, to the cloud while retaining core systems in-house. It also lets the company work with vendors on burst capacity during periods of peak demand, such as major sporting events.

“We’ve got IBM, Oracle and Dell as partners and work on a utility model and use a hybrid cloud model to handle peak loads, instead of building up our on-premises infrastructure for one day out of 365," Noonan said.

“We try to make sure app content is distributed, so while our core betting system is in Darwin, the content for the apps is hosted locally in a hybrid cloud. So graphics are hosted out of our hybrid cloud, while real-time data and the core customer transactions are handled by our back end in Darwin.”

Noonan said the company planned out its capacity needs and carried out a number of test runs ahead of the Spring Racing Carnival.

“One of the things in managing that volume is we know what to predict in terms of transactions per minute, and four weeks prior we carried out a dry run," he said.

“We model what demand will look like, and then go 20 percent higher to have extra capacity if, for example, a competitor goes down.”

Complex number crunching has long been a vital facet of the wagering industry, where accurately predicting betting odds is a fundamental task.

“We’re moving to the cloud (private and public) for data analytics. Currently, we use a mix of on-premises and public cloud for out data and models, depending on the complexity of the project,” Noonan said.

“We’re looking at vendors at the moment for public cloud, but we currently use AWS. Longer term, I suspect we’ll use multiple providers.

“We’ve gone with a best-of-breed approach for BI. We’ve made a significant investment in a customer data lake, processing, transformations and data mining, using multiple vendors and technologies, the main ones are GAP, Tableau, Greenplum, Talend, MapR, Hadoop and Python.

“The big advantage with the cloud is there is a huge increase in capacity and throughput. Computations that took five to 20 days on premises take less than three hours in the cloud.”

A highly competitive industry

The other major challenge faced by Noonan is the hyper-competitive nature of the wagering industry, and the resulting need to continually offer new products to customers.

With the average punter using at least two wagering apps on their smartphone, a company such as Sportsbet can’t afford to be late to market with new features.

It’s a challenge the company has met by adopting agile development methodologies.

“From a people point of view, we wanted to create a culture where people can develop, grow and rapidly make a contribution. Our business is in a highly competitive environment, where there’s a need for more products more often," Noonan said.

“We wanted to increase productivity, scale our technology platforms and decrease time to market. So we looked at what would be the most optimal way and it became clear, for customer outcomes and to retain and attract talent, that we had to go agile.

“Previously, we used a waterfall methodology where everything is scoped out ahead of time, and the release cycle was six to nine months. We’ve cut that down to two to four weeks.”

To help staff transition to the new approach, Noonan’s secret was to create an in-house agile ‘training academy’ during the early stages of the move.

“We’ve taken up agile processes in the past nine months. My expectation was that it would take 18 to 24 months to roll it out across all the teams – it was amazing to see the teams embrace the new agile ways of working," he said.

“We created an [internal] IT academy with agile projects in mind. In a four-week cycle we taught staff ways of working. Rather than have to pull people were in, people were drawn to it – they wanted to learn.

“Over time, the academy became redundant – staff began self-teaching within six to nine months.”

Agile structures

Self-organised teams made up of staff from across a range of different competencies work on a single feature at a time.

“We have what we call ‘feature teams’ working on projects. Each of these feature teams has IT ops, testers, scrum masters, project managers, UI, user experience, developers and engineers working co-located on each project or product," Noonan said.

“We have customer-facing project teams working on analytics, infrastructure, security and features, as well as internal IT project teams. The split is around 70 percent working on customer-facing projects and 30 percent on internal IT projects.

“Most of our core internal systems are quite new—they’ve been deployed within the past five years – and we use a SaaS model, so there’s less need for internal projects. Most of our internal project teams are focusing on analytics and data warehousing.”

Multitasking

The new structure allows for more than one feature to be worked on as part of an overall project at any given time.

“There’s one team per feature, and in some cases there are multiple teams per project. So for the punters club, for example, we have two teams working on different features, with very clear boundaries," Noonan said.

“There might be a third team that pivots across the portfolio depending on our priorities, depending on where we need more product or for a project to move faster.

“For example, there was a lot of work before the Spring Racing carnival on settlements, response times and the like.”

Future vision

Noonan and team are working to deliver on a three-year technology strategy that for customers focuses on an improved mobile experience, supported by use of shared services, data analytics, security, and APIs.

Internally, it targets people and culture and global ways of working.

Noonan said he was constantly on the lookout for new management insights.

“I travel to Silicon Valley twice each year and look at both leading companies and start-ups to see how they are working and how we can take best practice to continue to improve and grow our talent. That includes things like culture, engineering practices, automation and agile," he said.

“I’m passionate about making this the best time in our people’s careers."