Analysis: Will HP's data deduplication strategy pay off?

ANALYSIS: HP is promising big things for its StoreOnce deduplication software, but it may have a tough time standing out from its rivals.

Deduplication technology has soared in popularity in recent years, as companies seek to make better use of existing infrastructure by eliminating redundant storage of identical files. According to analyst group IDC, 60 percent of large enterprises have implemented deduplication or plan to this year.

At its TechForum 2010 event in Las Vegas, HP announced its own foray into deduplication with StoreOnce, an HP-developed suite of software designed to offer deduplication functions across a range of platforms.

That makes HP a relatively late entrant to the deduplication market, but SVP of the company's StorageWorks division David Roberson [pictured] argues that current solutions don't provide a unified experience.

"It's not easy, it's not seamless and the data has to be expanded and contracted every time you move it from one platform to another," he said. "This is going to change the way deduplication is done."

Rather than buying in deduplication technology (a strategy pursued by EMC with its acquisition of Data Domain, HP Labs built StoreOnce from scratch.

The software includes a number of new technologies designed to improve deduplication performance, including sparse indexing, intelligent block size management and a reduction in disk fragmentation.

"The key problem with any deduplication technology is you've got a lot of data you've got to index, and your index is also really big. You either use a lot of memory or reduce performance by pulling data from disk," explained Alistair Veitch, director of storage and information management platforms at HP Labs. "Sparse indexing is a way of partitioning the index very efficiently so that when you're seeing new data you only have the pieces of data in memory that you need."

"We use a very small block size which gives us a lot more deduplication efficiency. There's a lot of work gone into how we lay things out across the disk so we reduce fragmentation. And if we do have to go to disk, we get good performance there."

While HP is talking up the potential for StoreOnce to operate across the enterprise and says it will be seen across its range over the next year, its first incarnation will be in a more conventional disk-based backup environment (the 4312) that, at first glance, isn't notably different from the deduplication appliances released by EMC, NetApp and other storage vendors. That might make it hard for it to stand out in the market.

HP will rely on its large installed base and a planned expansion of the software's platform base. 

HP will also promote integration of StoreOnce with third-party software. "The first backup software we're going to put this on is our own HP Data Protector software," said Tom Joyce, VP of marketing strategy for StorageWorks. "We expect to see this on other company's backup software over time."

Convincing rivals to adopt an HP-developed technology might be tricky, even given HP's current dominance of the general server market. In a tacit recognition of its late-to-market status, the HP package supports Symantec's widely adopted OpenStorage API.

While deduplication can help businesses use existing technology more efficiently, its appeal for enterprise vendors stems in large part from its rapid uptake and potential to increase margins.

Not everyone sees value in deduplication, whatever the platform. Earlier this year, IBRS analyst Dr Kevin McIsaac described it as a "band aid solution" and said that companies would be better off concentrating on developing consistent data management strategies rather than trying to massage existing data.

Disclosure: Angus Kidman travelled to TechForum as a guest of HP.