Analyst: Don't be duped by de-dupe

De-duplication described as a "band-aid" solution.

De-duplication, one of the most hyped storage technologies of the past two years, is a "band-aid" solution for organisations with lax IT policies, according to one of Australia's leading enterprise IT analysts.

IBRS analyst Dr Kevin McIsaac said that while there are some use cases for de-duplication - which aims to identify multiple redundant copies of the same file in a given storage system, deleting copies and leaving a pointer to one single copy to save the organisation disk space - it is not the panacea portrayed by some storage vendors.

In many cases, the analyst said, organisations would be better off improving their data management processes to avoid duplication of the data in the first place.

McIsaac told iTnews that the technology assumes that organisations have multiple redundant copies of the same data on their network.

Such a scenario is often caused by ineffective data management policies within the organisation, he said.

McIsaac said a common use case made for de-duplication is for email storage. The suggestion is that users within an organisation often receive the same emails, and that multiple copies of the same email data and attachments stack up.

But McIsaac noted that recent versions of the world's most popular corporate email suite, Microsoft Exchange, already has "single instance storage" capabilities natively.

Multiple users may have the same email in their inbox, but that data is stored only once at the back-end.

Duplication is created when users copy an email or attachment to their network drives, usually because the organisation have implemented inbox quotas or they lack a document management system.

"Sure, you can apply de-duplication as a band-aid, but there is a lot more value in implementing automated email archiving and document management." he said. "Using these approaches liberates the user from inbox limits, enables people to find information more quickly and can supports compliance and e-discovery processes. "

McIsaac also notes that the de-duplication of virtual machines comes hand-in-hand with a performance compromise, and that using de-duplication when backing up files across the network may itself be redundant if the organisation already backs up its data on an "incremental" basis (ie only backs up those files that have changed on any given day).

McIsaac's views run counter to the investment strategies of the world's largest storage vendors. Last year EMC and NetApp engaged in a bidding war for de-duplication specialists Data Domain - a race EMC won. Now EMC offers Data Domain branded appliances through its own channels and via OEM agreements with the likes of Dell.

 "Part of the issue is that storage vendors continually need to add new features to prevent the commoditisation of their products, and de-duplication is the current hot feature," McIsaac said.

"However in most contexts, de-duplication is not a strategic platform - it is a band-aid solution and customers should use it and value it on that basis" .