Woodside Energy sets up a unified enterprise data platform

Woodside Energy has used the past few years to set up Snowflake as a unified data platform for both structured and time-series data, moving off platforms previously assembled from AWS components.

The liquefied natural gas (LNG) producer described its migration to Snowflake at the vendor’s world tour summit in Sydney last month.

Digital platform operations excellence lead Dameon Russell said Woodside had “been working with data platforms since about 2017”.

“By 2021, we had two pillars of a data platform that we were actually supporting: one was a relational data lake and the other was a time-series platform,” he said.

“Both of them were built on a variety of AWS services, and both of them had the same sort of technical problems.”

Russell said there were “few” internal resources “that really understood how these things worked deeply and were able to make modifications”; that “platform changes were fragile; security and governance was really a second thought”; and there was “little observability” around the existing platform, all of which contributed to operational complexity.

“As all this was going [on], people [internally] were saying: ‘We've got to be a data-driven solution organisation’, yet people didn't really trust the platforms and they didn't really trust the data,” Russell said.

Data technology platform lead Rohan Davies said “a new data platform project” was stood up “to address … concerns” around governance and security, and with “a view to minimise the total cost of ownership.”

“We wanted a modular architecture, so no longer a monolith, that would allow us to swap out components as we saw fit, and have a really best-of-breed approach,” Davies said.

The gas producer selected Snowflake as the core of the new data platform, initially for the role-based access control (RBAC) and security capabilities, as well as its manageability and configurability.

“We introduced separate tools for [data] ingestion, for transformation and [data] catalogue, and we [also] introduced DataOps, so all of the changes we make are … through Git repos. That allows us to automate the test and deployment process,” he said.

“That whole process of allowing us to have code drive the platform was a real win for us.”

The company re-mapped source and consumption integrations for the new platform, but did not have a "‘big bang’ cutover”. 

Instead, Woodside progressively established Snowflake as the new relational -- or structured -- data platform for the organisation across 2023 and 2024.

“[In] 2024, we [were] almost completely off our legacy data system. We had a couple of systems left,” Davies said.

“We are now sitting at just over 95 million records a day ingested, just under 2500 consumable data assets, and sitting at just over 250,000 queries a day - [and] people are starting to see the potential.”

“Around the same time Snowflake was kicking off” for structured data, Russell said that Woodside also started reviewing options for managing its time-series data: primarily data readings taken from its industrial plant to show “the state of the plant”, and determine - often using subject matter experts - whether the readings were healthy or unhealthy

These readings were increasing in volume and frequency, particularly as Woodside invested in additional gas projects.

Additionally, the company wanted to increase automation of this data analysis.

This part of the organisation initially considered 15 data platforms, whittling that down to three before also going with Snowflake.

Snowflake’s selection in part was made based on testing using an “aspirational” workload; the increased volume could be handled relatively easily with Snowflake by tweaking the cluster size and configuration, but was found to require re-architecture with other platforms.

Russell said that Snowflake is now supporting “500,000 production queries a day, hitting our time series data sets.”

While acknowledging criticism of Snowflake not specifically being a time-series data platform, Russell said it had “time-series features”, which Woodside said had proven “capable of delivering this workload for us”.

The company has continued to advocate for the expansion of these capabilities directly with the vendor.

“We're always pressing Snowflake, I guess, for more time-series functions,” Russell said.

Woodside has also started using Snowflake Gen-2 virtual warehouses to underpin the time-series data workloads, which has allowed it to decrease cluster sizes and improve query performance.

Davies said that in 2025, Woodside now operates “a unified data platform” for relational and time-series data.

“We're now an integral part of digital initiatives supporting all areas of the business,” he said, adding that data owners and consumers across Woodside recognised the value of the platform to their respective domains.