Questions tagged [deduplication]
For questions where a task is to be applied to only one instance of multiple copies of data (files or blocks of data on a filesystem, or strings in a text), or where duplicates of the first such instance are to be ignored for space/time saving purposes.
78
questions
1
vote
2
answers
4k
views
Identify unique records on CSV based on specific columns [duplicate]
If the input is
foo,bar,baz
bar,baz,qux
qux,quux,baz
bar,foo,qux
waldo,fred,garply
the output should be
foo,bar,baz
bar,baz,qux
waldo,fred,garply
As you can see, records are deduplicated based on ...
- 13
-2
votes
1
answer
393
views
Command to delete duplicate files from current directory [duplicate]
I have a directory with several duplicate files, created by a program. The duplicates have the same name (except for a number), but not all files with the same name are duplicates.
What's a simple ...
- 574
8
votes
1
answer
2k
views
What does a rmlint's "clone" for btrfs do?
I was reading the rmlint manual, and one of the duplicate handlers are clone and reflink:
· clone: btrfs only. Try to clone both files with the BTRFS_IOC_FILE_EXTENT_SAME ioctl(3p). This will ...
- 9,492
13
votes
5
answers
33k
views
How to find duplicate lines in many large files?
I have ~30k files. Each file contains ~100k lines. A line contains no spaces. The lines within an individual file are sorted and duplicate free.
My goal: I want to find all all duplicate lines across ...
- 242
4
votes
6
answers
3k
views
How to append multiple lines to a file, if these lines doesn't exist in that file?
How to append multiple lines to a file, if these lines doesn't exist in that file?
For example, to add multiple global aliases to /etc/bash.bashrc I use an heredocument:
cat <<-"BASHRC"...
1
vote
1
answer
211
views
script for moving kept files deleted with fdupes or other deduplicating package
After I recovered my files with PhotoRec, I ran fdupes $ fdupes -rdNI --sameline /home/user > fdupes.txt. I couldn't use the option -d because round about 1.6 million files in my system have been ...
- 697
1
vote
1
answer
527
views
How can I tell which copies of a file are COW copies?
I have a ZFS system connected to a macintosh over NFS. The system is nearly full, but I have found a large number of duplicate files with jdupes.
I don't know how these copies were made, it's ...
- 9,492
6
votes
2
answers
9k
views
How does tar deal with hardlinked files? [duplicate]
I have a 2.5 TB of data that I want to put in a 2TB hard drive to mail somewhere. It's not hopeless, as a very large fraction of the data consists of duplicate files. I am considering using jdupes ...
- 9,492
6
votes
4
answers
3k
views
cp --reflink=auto for MacOS X
cp --reflink=auto shows following output for MacOS:
cp: illegal option -- -
Is copy-on-write or deduplication supported for HFS? How can I COW huge files with HFS?
- 285
9
votes
1
answer
2k
views
Make tar (or other) archive, with data block-aligned like in original files for better block-level deduplication?
How can one generate a tar file, so the contents of tarred files are block-aligned like in the original files, so one could benefit from block-level deduplication ( https://unix.stackexchange.com/a/...
- 14.3k
1
vote
1
answer
71
views
Mountable deduplicating container
I want a container that I can mount which dedups files within it.
Basically, I'd like to have a simple backup solution where I mount it, copy my home folder with the name "Home (Feb 9)" and it only ...
- 11
17
votes
1
answer
6k
views
How to duplicate a file without copying its data with btrfs?
I have no experience with btrfs, but it's advertised to be able to
de-duplicate files.
In my application, I'd need to duplicate whole directory trees.
From what I learned, btrfs only de-duplicates ...
- 1,123
3
votes
2
answers
152
views
Find files recursively, but choose largest from among those with duplicate names
Given a nested directory structure containing various files, I would like to find all the files within it, but where there are multiple files with the same name, I'd like to return just the largest ...
- 385
2
votes
2
answers
192
views
transfer many similar files over ssh
A service on a linux server is only able to do full backups, where each backup is a .tar archive (no compression). Many contents of the archive do not change from day to day. Each .tar file size is ...
- 2,818
5
votes
1
answer
1k
views
Are tars deduplicatable at the block level?
Quite simply, when a tar file is made on disk, would the extents be deduplicatable with extents inside and/or outside of the tar? I am asking in the theoretical sense, so if the extents of data are ...
- 351
4
votes
2
answers
2k
views
Is `--reflink=auto` safe to set as default for cp?
I am currently a user of BTRFS and would like to take advantage of the CoW such that when files are copied on the same btrfs filesystem, they are automatically deduplicated by reusing the existing ...
- 351
2
votes
1
answer
400
views
Deduplication semantics with btrfs - meta-data differs, file data identical
So I read this Wiki article on deduplication with btrfs. However, it doesn't describe the semantics followed by btrfs deduplication.
Assume you have a dozen files. They all contain identical data, ...
- 16.6k
5
votes
1
answer
2k
views
How to get tar to hardlink identical content on-the-fly during archive creation?
How can I get tar (or any other program that is commonly available on Linux, e.g. pax) to hardlink duplicate content on-the-fly during archive creation?
That is, I'd like to avoid hardlinking upfront ...
- 16.6k
5
votes
3
answers
2k
views
deduplication of lines in a large file
The size of the file is 962,120,335 bytes.
HP-UX ******B.11.31 U ia64 ****** unlimited-user license
hostname> what /usr/bin/awk
/usr/bin/awk:
main.c $Date: 2009/02/17 15:25:17 $Revision: ...
- 1,229
3
votes
2
answers
7k
views
Deduplication tool for rsync?
Is there a Linux / Unix tool that does de-duplication similar to the way Veeam does it before compressing and sending a backup with rsync to a mirrored location?
- 4,625
9
votes
5
answers
4k
views
Remove duplicate lines from a file that contains a timestamp
This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp.
Is it possible to tell awk to ...
- 3,253
3
votes
1
answer
682
views
How to evaluate if it's worth using deduplication?
I have a partition where I am considering to use deduplication.
For the profile of it's data I think it will be a good choice. Still, before doing it, I would like to evaluate the impact in a more ...
- 291
11
votes
2
answers
4k
views
How to find data copies of a given file in Btrfs filesystem?
I have deduplicated my Btrfs filesystem with bedup, so now all duplicate files (above a certain size) are "reflink" copies.
Is there any way to see, given a filename, what other files are the same ...
- 1,164
1
vote
1
answer
1k
views
Can a overlay/union file system be used as makeshift deduplication?
Given a union file system (mhddfs, unionfs, aufs) would it save me anything if I overlaid a writable file system over one with a bunch of ISO files, say Ubuntu 10.04, and then use zsync to update the ...
- 16.6k
12
votes
3
answers
3k
views
Are there any deduplication scripts that use btrfs CoW as dedup?
Looking for deduplication tools on Linux there are plenty, see e.g. this wiki page.
Allmost all scripts do either only detection, printing the duplicate file names or removing duplicate files by ...
- 1,164
2
votes
1
answer
2k
views
Synchronize directory tree with deduplication?
I have two Linux servers that have a large amount of data (1TB+) that need to be synchronized over a slow connection (100 KB/s).
A lot of the data overlaps, but are in different locations.
I would ...
- 343
9
votes
2
answers
2k
views
Deduplication on partition level
What are available solutions for block level or more detailed deduplication ?
There are file-based ones - with "Copy-On-Write" approach.
I'm looking for block level "copy-on-write", so I could ...
- 14.3k
203
votes
20
answers
74k
views
Is there an easy way to replace duplicate files with hardlinks?
I'm looking for an easy way (a command or series of commands, probably involving find) to find duplicate files in two directories, and replace the files in one directory with hardlinks of the files in ...
- 8,458