The tag has no usage guidance.

learn more… | top users | synonyms

6
votes
1answer
59 views

Make tar (or other) archive, with data block-aligned like in original files for better block-level deduplication?

How can one generate a tar file, so the contents of tarred files are block-aligned like in the original files, so one could benefit from block-level deduplication ( ...
1
vote
1answer
25 views

Mountable deduplicating container

I want a container that I can mount which dedups files within it. Basically, I'd like to have a simple backup solution where I mount it, copy my home folder with the name "Home (Feb 9)" and it only ...
10
votes
1answer
131 views

How to duplicate a file without copying its data with btrfs?

I have no experience with btrfs, but it's advertised to be able to de-duplicate files. In my application, I'd need to duplicate whole directory trees. From what I learned, btrfs only de-duplicates ...
3
votes
2answers
64 views

Find files recursively, but choose largest from among those with duplicate names

Given a nested directory structure containing various files, I would like to find all the files within it, but where there are multiple files with the same name, I'd like to return just the largest ...
2
votes
2answers
55 views

transfer many similar files over ssh

A service on a linux server is only able to do full backups, where each backup is a .tar archive (no compression). Many contents of the archive do not change from day to day. Each .tar file size is ...
4
votes
1answer
168 views

Are tars deduplicatable at the block level?

Quite simply, when a tar file is made on disk, would the extents be deduplicatable with extents inside and/or outside of the tar? I am asking in the theoretical sense, so if the extents of data are ...
2
votes
1answer
334 views

Is `--reflink=auto` safe to set as default for cp?

I am currently a user of BTRFS and would like to take advantage of the CoW such that when files are copied on the same btrfs filesystem, they are automatically deduplicated by reusing the existing ...
2
votes
1answer
122 views

Deduplication semantics with btrfs - meta-data differs, file data identical

So I read this Wiki article on deduplication with btrfs. However, it doesn't describe the semantics followed by btrfs deduplication. Assume you have a dozen files. They all contain identical data, ...
4
votes
1answer
230 views

How to get tar to hardlink identical content on-the-fly during archive creation?

How can I get tar (or any other program that is commonly available on Linux, e.g. pax) to hardlink duplicate content on-the-fly during archive creation? That is, I'd like to avoid hardlinking upfront ...
2
votes
3answers
365 views

deduplication of lines in a large file

The size of the file is 962,120,335 bytes. HP-UX ******B.11.31 U ia64 ****** unlimited-user license hostname> what /usr/bin/awk /usr/bin/awk: main.c $Date: 2009/02/17 15:25:17 $Revision: ...
3
votes
2answers
947 views

Deduplication tool for rsync?

Is there a Linux / Unix tool that does de-duplication similar to the way Veeam does it before compressing and sending a backup with rsync to a mirrored location?
8
votes
5answers
1k views

Remove duplicate lines from a file that contains a timestamp

This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp. Is it possible to tell awk to ...
3
votes
1answer
170 views

How to evaluate if it's worth using deduplication?

I have a partition where I am considering to use deduplication. For the profile of it's data I think it will be a good choice. Still, before doing it, I would like to evaluate the impact in a more ...
3
votes
2answers
2k views

How to find data copies btrfs

I have deduplicated my btrfs filesystem with bedup, so now all duplicate files (above a certain size) are "reflink" copies. Is there any way to see, given a filename, what other files are the same ...
1
vote
1answer
368 views

Can a overlay/union file system be used as makeshift deduplication?

Given a union file system (mhddfs, unionfs, aufs) would it save me anything if I overlaid a writable file system over one with a bunch of ISO files, say Ubuntu 10.04, and then use zsync to update the ...
8
votes
1answer
2k views

Are there any deduplication scripts that use btrfs CoW as dedup?

Looking for deduplication tools on Linux there are plenty, see e.g. this wiki page. Allmost all scripts do either only detection, printing the duplicate file names or removing duplicate files by ...
2
votes
1answer
1k views

Synchronize directory tree with deduplication?

I have two Linux servers that have a large amount of data (1TB+) that need to be synchronized over a slow connection (100 KB/s). A lot of the data overlaps, but are in different locations. I would ...
8
votes
1answer
678 views

Deduplication on partition level

What are available solutions for block level or more detailed deduplication ? There are file-based ones - with "Copy-On-Write" approach. I'm looking for block level "copy-on-write", so I could ...
80
votes
17answers
25k views

Is there an easy way to replace duplicate files with hardlinks?

I'm looking for an easy way (a command or series of commands, probably involving find) to find duplicate files in two directories, and replace the files in one directory with hardlinks of the files in ...