Amaysim rebuilds analytics on AWS

Fast-growing telco Amaysim has overhauled its analytics architecture on Amazon Web Services to harness streaming data and more readily integrate information from various acquired companies.

The mobile virtual network operator (MVNO) has expanded swiftly through acquisitions such as AusBBS and Vaya in 2016 and Click Energy in 2017, diversifying into fixed broadband and electricity while growing 29 percent in the most recent financial year to $326.7 million.

This M&A journey led to multiple data silos from each legacy business, creating challenges for Amaysim's business intelligence team.

Amaysim migrated its infrastructure to AWS three years ago and is currently transitioning to a microservices architecture.

The business intelligence team is now standardising Amaysim's analytics environment on Amazon's suite of analytics tools, BI director Ferhod Sarmed told iTnews on the sidelines of the Big Data & Analytics Innovation Summit in Sydney.

Legacy data is ingested from three distinct sources: traditional hosted databases such as PostgreSQL, SQL or smart meter reading from Click Energy; flat file formats, such as SIM card provisioning reports by its third-party distributor; and, increasingly, streaming data.

Sarmed sees the future in streaming data, which includes app usage, web traffic and IoT information. "We want to break our reliance on databases and flat files and start to do everything through streaming and APIs."

It's early days for streaming data, which Amaysim collects, processes and analyses using AWS Kinesis Firehose. 

Once ingested, data is stored in its native file format in AWS S3 buckets, which cost a fraction of the price of storing data in an AWS Redshift data warehouse.

"With the Redshift cluster we have got, we probably hold around 30 percent of the available information about the business; most of it is transactional usage, so you can look at finance reports and customer information," he said.

"At any one given point in time, we use about 20 percent of the data in Redshift. So for me there is an economical reason there [to use S3 buckets instead]."

The telco is using the new Amazon Redshift Spectrum service, which allows customers to run Amazon Redshift SQL queries against data in S3 buckets. Amaysim also uses AWS' pay-as-you-go serverless interactive query service, Athena.

Pilots AWS Glue

Amaysim is an early adopter of the new AWS Glue tool, which is a fully managed extract, transform and load (ETL) service and replaces Apache Drill at the telco.

It was one of five Australian companies to pilot Glue, which has just been made generally available in the US, he added.

Sarmed said the new analytics architecture has helped the team avoid significant work to integrate data from the businesses Amaysim acquired over the past two years.

"One business is working on Oracle, one is on SQL Server and so on and so forth," said Sarmed.

"Traditionally you would need this crazy ETL to mash it into one final warehouse technology, but with this we can put them in their native format into S3 buckets, then Glue can read them. We miss a step of trying to build complex ETL to integrate different systems.

"There is still lot of integration from a production sense, but we're having a fun time in analytics at least," Sarmed said.