Coles sets up standard data streaming platform groupwide

Coles Group has stood up an enterprise-wide data streaming platform over the past two years, simplifying and standardising the way it handles real-time operational data.

Simon Bedford.

The retailer first spoke about its intentions to shift reliance from batch- to real-time data processing back in 2023.

It appears this has been made simpler in recent times by establishing Confluent Cloud - a managed Apache Kafka service - as a standard technical foundation for the effort.

Speaking at the Data Streaming World Tour event in Melbourne, principal engineer Simon Bedford said that in the absence of an enterprise standard, engineering teams previously made their own choices around data streaming stacks.

“All the different teams had their own eventing stacks, all tied to one another,” Bedford said.

This architecture did not scale well as Coles’ real-time data ambitions grew.

“We had many different isolated eventing stacks. We had a lot of data silos, inconsistent models of access, no standard way of working, [and] there was duplication of data,” Bedford said.

“It was really difficult to discover any data in this model because stuff is everywhere. A lot of the time you just had to know the right person to know if the data was there [or to get things done.”

“We had a lot of problems with this architecture.”

These problems ranged from difficulty enforcing standards, controls and governance, to additional costs.

Recognising the challenges, the retailer changed direction and elected to set up a single “enterprise event platform”.

“We treat our platform as a product internally. We sort of treat it as like an internal SaaS,” Bedford said.

“Our customers are our developers and our architects - and the adoption of our platform depends on [the] developer experience. 

“If the devs don’t trust the platform, or they don’t like it, they’ll just work around it. 

“The platform needs a clear roadmap. We need a vision for the platform and [to] work towards it. Without a vision, the platform is more of a utility rather than a strategic enabler, and we want our platform to be a strategic enabler.”

'Not just Kafka'

While the platform is technically powered by Confluent Cloud, Bedford said it was more than its underlying infrastructure.

“It’s not just Kafka,” he said. 

“There’s more than that. There’s self-service in our platform - self-service is embedded. 

“We wanted [to implement] automation, schemas, [and] compatibility checks - all the good things we can get through a platform.

“We wanted to enable policy-as-code across the platform to ensure people are aligned with the governance, and then we get governance that actually scales. We [also] wanted to enable the [data] discovery side of things.

“Confluent provides us with a lot of the tools to get there but [we] need to build on them.”

Bedford said that engineers can self-serve their onboarding to the streaming platform. Once there, they can then create their own topic - the fundamental unit of organising data in Kafka.

“We put governance over the top of [topic creation] to make sure it aligns to standards. It’s not the wild west - [there are] policy-as-code procedures in place,” he said.

That governance begins with topic naming.

“Topic naming isn’t cosmetic - it defines governance, security, accountability [and] cost management across the platform,” Bedford said.

For example, Bedford said that naming “allows for cost attribution. We can determine what topic drives costs, so we can … tie it to initiatives we’ve been through.”

“The other benefit around naming upfront is you get consistency - it improves your discoverability. When people go looking they know where to find things. Things are clear and predictable at scale.”

Developer experience improvement

The team managing the event streaming platform continued to focus on developer experience to grow usage and adoption, Bedford said.

“The platform team is really positioned around removing friction from onboarding, to enable features, to introduce capabilities to improve productivity.

“When Confluent comes to us and says, ‘Hey, we’ve got a new feature’, we want to enable that internally as well. 

“We want to have continuous developer experience improvement.”

Business benefits

The result of all of this is a “structured, governed platform” for data streaming at Coles.

Bedford said the broader Coles business also benefitted from the platform.

“Some of the business outcomes we’ve achieved through it are probably more cost oriented,” he said.

“We have faster time-to-market as a whole. 

“With our self-service integration, it saves people’s time from a significant endeavour to work out what to do and how to get access [to data] and go through manual approvals and chasing every manager for days to approve a ticket, to click a button and you get access. 

“Developer time is significantly saved. This helps lower overall integration costs as a result. 

“We’re [also] getting cheaper new integrations over time. So now that we have data on a  platform and we’ve got a lot of topics and we’ve got a reasonable level of quality around it, we can now get reusable patterns out of it. Go ask the producer for access to data, and they can do that through the platform.”

He added that access to real-time data could also lead to faster operational decisions being made.

Some of this will come with future work to harness the platform, including using the data to power analytics workloads, and feeding the data to AI models.

“We’ve got the operational data - how can we feed it into AI systems safely, supporting things like forecast optimisation and anomaly detection?” Bedford said.