Kmart Australia changes the make-up of its data platform

Kmart Australia has swapped out several core AWS components powering the retailer’s ‘next generation’ data platform and replaced them with a combination of Snowflake and Microsoft Power BI.

The technology change was revealed during a presentation at Snowflake’s retail and CPG data analytics forum that also shed new light on the retailer’s growing array of use cases for data analytics.

The switch to Snowflake and Power BI was characterised as “a minor change” by cloud data architect Vivek Vijay Sinha, at least from an architectural standpoint.

“The entire pattern of the architecture ... looks very similar to what we had,” he said, showing 2019 and 2020 diagrams as a before-and-after.

However, where AWS services like S3, Athena and Quicksight once powered key elements now sits Snowflake and Power BI.

It’s a marked change from late 2019 when Kmart Australia unveiled its near full-stack AWS reliance at the cloud company’s flagship re:Invent conference in the United States.

The data platform has also been renamed ‘Sophia’, which was first revealed by KAS Services, Kmart Group’s India-based innovation and technology hub, earlier this year.

Data goals

Kmart Australia’s head of data and analytics Vineet Mehta told the Snowflake conference that the retailer needed a strong data platform to meet its ambitions for customer service and supply chain optimisation.

“We're always looking to understand our customers better and meet their rapidly rising customer expectations in the retail landscape. Our ambition is to actually be able to offer a personalised experience to each and every Kmart customer,” Mehta said.

“Our goal is to use the best in class data, machine learning and AI to know our customers better, anticipate their needs, and serve them in a way where everyday living is brighter. 

“In parallel, we are also transforming our supply chain to improve product availability and provide a flexible omnichannel customer experience of shopping anytime, anywhere.

"We are transforming our supply chain to be data-driven, supported by the latest in technology from RFID to robotics to artificial intelligence.

“Data and analytics is a key strategic enabler and the cornerstone for these capabilities. We needed a foundational state-of-the-art data platform that can be the backbone of this orchestration.”

Mehta said Kmart Australia had spent the past year “partnered with Snowflake to build a modern cloud data platform.” 

“The Sophia data platform helps us provide the scale and speed needed to build strong data foundations and data services,” he said.

“The platform is used today across the enterprise by business users and technologists in various functional roles, and helps us enable insights and analytics.”

Addressing scale, latency challenges

Sinha noted that when Kmart Australia first started its journey to stand up a new data platform in 2019, “we were very heavy on AWS”.

“We were already utilising almost all the services of AWS, so we thought of stitching all the AWS services [together to] build a data platform, which is based on cloud,” he said.

However, Sinha said, the retailer ran into challenges with the initial make-up of the data platform.

“The problem was as the data [volumes] started growing, with time we started seeing certain challenges with this architecture,” he said.

In addition to scalability complaints, Sinha said there was unacceptable latency when generating reports, and that “query processing and storage was very costly”.

There were other challenges listed - around dataset maintenance and the layering of governance, security and compliance rules on top of the platform - though no specific detail was provided around any of the identified challenges. 

Sinha said the challenges were viewed internally as “red flags” that the data platform, as it was made up, was “not going to be a sustainable platform ... to go on a future journey” with analytics.

“We thought, ‘we need to really evolve from this and take the next step in our data journey’,” he said.

“So when 2020 came, we changed a few things. We changed the AWS Athena, S3 buckets and Quicksight to a combination of Snowflake and Power BI.

“This single change turned out to be a real gamechanger for our entire data platform.”

Data handling improved, processing could be scaled up and down faster, and caching and acceleration features in Snowflake helped resolve the latency issues for reporting, Sinha reported.

Kmart also used federated authentication features to enable “secure data exchange” between affiliate groups.

Sinha provided some statistics on the size of the Sophia data platform, noting it encompasses 60+TB of data, and handles about 30,000 queries a day.

“Hopefully we'll reach petabyte scale,” Sinha said. “We're still confident Snowflake would support that.”

The data platform comprised 30+ “virtual [data] warehouses that are supporting various business functions and developers”, he noted. Over 300 staff interact with the data platform.

He added that Kmart has “close to 1000+ dashboards and reports, which are on Power BI, getting powered by the engine within Snowflake.”

On the use case front, Sinha said that Snowflake is powering Kmart Australia’s data lake operations as well as “data marts” stood up for specific reporting or machine learning applications.

In addition, it is being used to enable secure sharing of data with affiliates, as well as for “a new concept called business workspace” where “anyone and everyone in the organisation who knows SQL can get a copy of the data that we have already ingested and stored, and they can create their own insights.”

“This has actually evangelised the culture of data in the organisation,” Sinha added.