13 Facts About Data deduplication

In computing, data deduplication is a technique for eliminating duplicate copies of repeating data.

Whereas compression algorithms identify redundant data inside individual files and encodes this redundant data more efficiently, the intent of deduplication is to inspect large volumes of data and identify large sections – such as entire files or large sections of files – that are identical, and replace them with a shared copy.

In computer code, Data deduplication is done by, for example, storing information in variables so that they don't have to be written out individually but can be changed all at once at a central referenced location.

Storage-based data deduplication reduces the amount of storage needed for a given set of files.

In-line network data deduplication is used to reduce the number of bytes that must be transferred between endpoints, which can reduce the amount of bandwidth required.

However, certain vendors with in-line Data deduplication have demonstrated equipment which is able to perform in-line Data deduplication at high rates.

The Data deduplication process is transparent to the users and backup applications.

Source deduplication can be declared explicitly for copying operations, as no calculation is needed to know that the copied data is in need of deduplication.

Target deduplication is the process of removing duplicates when the data was not generated at that location.

The Data deduplication process is intended to be transparent to end users and applications.

Commercial Data deduplication implementations differ by their chunking methods and architectures.

The second reason why deduplication is applied to secondary data, is that secondary data tends to have more duplicate data.

Data deduplication has been deployed successfully with primary storage in some cases where the system design does not require significant overhead, or impact performance.