Optus chasing "near real-time" data as regulatory pressure looms

Optus has moved to pull information from legacy customer-facing applications stored in an Oracle database faster to help it meet regulatory and compliance obligations.

The carrier’s associate director of design and delivery Anupam Das said that the telco had been using a third-party connector and streaming platform to pipe the data into its Databricks analytics platform in “near real-time”.

Speaking at a Databricks customer event, Das said that the carrier is using Confluent’s Oracle CDC (Change Data Capture) and LogMiner in conjunction with its Kafka streaming platform to pull 400 data tables out of Oracle to be fed into Databricks.

“That's where we have the analytical use cases, the reporting use cases and, most important right now, our regulatory use cases, which means this data is very crucial in terms of accuracy,” Das said.

He said that the method had accelerated the process of preparing data for Optus leadership and other parts of the business from days down to hours.

The intent is to provide leadership with an accurate portrait of how the business has fared within the past day.

"The most important [driver] is the accuracy of the data, so that we can give it to our leadership and say, ‘OK, this is how Optus worked in the last 24 hours’. This is where we can see … Optus as a company needs to focus more.

"This was not possible two years back ... We had to wait for two or three days. Now it is within hours,” he said.

Beyond meeting regulatory and compliance obligations, the technology is also allowing Optus to track customer behaviour in its stores in near real-time, opening up marketing opportunities for the carrier such as product offers based on purchase behaviours.

Das said that it would also open up potential to apply more machine learning and AI to the data, rather than being locked in legacy siloes.

Unified data platform

Optus has recently completed a major transformation, standing up a unified data platform built by Optus and a consortium of Microsoft, Databricks and Tech Mahindra.

The UDP pulls together data from across its business into a data lakehouse on Azure, with Databricks providing the analytical and governance smarts to manage how it's used across the organisation.

Speaking to the iTnews Podcast at the time of the platform’s operational launch, Optus chief information officer Mark Potter said that transformation would happen in two phases.

The first was a lift-and-shift away from a data warehouse reaching end-of-life and the second, then expected to be concluded this year would see “two other very substantial data marts” consolidated into the UDP.

However, he also said that some data would remain outside the UDP only to be included on as needed basis.

“We’re driving a high degree of data consolidation, but we're not executing a strategy of one data mart for the entire organisation,” Potter said.

Das said that Confluent’s data streaming technology is already playing a “key role” in the transformation, which would continue for now.

“The tech stack will probably be simplified. The tech stack may change, but the bus layer, the moving the data into the data platform, that's where we are seeing [that] Confluent will play a significant key role,” Das said.