Questions tagged [deduplication]

For questions where a task is to be applied to only one instance of multiple copies of data (files or blocks of data on a filesystem, or strings in a text), or where duplicates of the first such instance are to be ignored for space/time saving purposes.

Filter by
Sorted by
Tagged with
1 vote
1 answer
24 views

Choosing the right block size for duperemove

I am trying to deduplicate a BTRFS filesystem with multiple subvolumes. Altogether, it holds around 3.5 TB of data, which I expect to be slightly more than half that size after deduping. I am mostly ...
1 vote
1 answer
174 views

Is there a way to consolidate (deduplicate) btrfs?

I have a btrfs volume, which I create regular snapshots of. The snapshots are rotated, the oldest being one year old. As a consequence, deleting large files may not actually free up the space for a ...
0 votes
4 answers
92 views

Filtering duplicates with AWK differing by timestamp

Given the list of files ordered by timestamp as shown below. I am seeking to retrieve the last occurrence of each file (the one at the bottom of each) For example: archive-daily/document-sell-report-...
-2 votes
1 answer
119 views

low-memory server with ZFS deduplication: can zram help?

Is it a good idea to use compressed memory if people want to save on expensive RAM when using ZFS with deduplication?
  • 31
-1 votes
1 answer
83 views

can swap help a low-memory ZFS server?

If we cannot buy many RAM, can we replace the "missing" RAM with higher amount of SWAP? Ex.: using a dedicated 512 GB SSD for SWAP instead of 512 GB RAM for ZFS with deduplication on an ...
  • 31
0 votes
0 answers
22 views

find duplicates inside archives

I have a huge archive folder and want to integrate that into a local backup folder and then eliminate duplicates before making a cloud backup of the whole backupfolder. The problem: the files in the ...
  • 1
0 votes
2 answers
46 views

Why do these two commands to write text-processing results back to the input file behave so differently?

I have a file authorized_keys and want to deduplicate the content, i.e. remove duplicate entries. I found two possible solutions to achieve this: Use cat and uniq, then redirect the output to the ...
  • 1
0 votes
1 answer
40 views

Why did sort -u removed duplicates only in pipe?

In this command, sort -u removed duplicates. curl https://en.wikipedia.org/wiki/Help:Special_page -s | grep -oP 'Special:\K[a-zA-Z0-9]*' | sort -u > special_page_names In this command, it didn't. ...
  • 115
1 vote
1 answer
159 views

How to let already copied files share fragments (reflink)?

I copy a file to a different XFS volume on daily basis as follows: # on monday cp --sparse=always /mnt/disk1/huge.file /mnt/disk2/monday/huge.file # on tuesday cp --sparse=always /mnt/disk1/huge.file /...
  • 145
0 votes
1 answer
333 views

lvmvdo doesn't deduplicate my data

I am installing lvmvdo con Debian 11.2 to store Proxmox vm disk, doing the following : 1) apt install -y build-essential libdevmapper-dev libz-dev uuid-dev git sudo libblkid-dev man vim dwarves dkms ...
1 vote
1 answer
48 views

Meaning of Deduplicated during Borg Create's Realtime Output

When the borg create command is used with the --progress argument, it output like this: 5.50 GB O 5.10 GB C 23.95 kB D 15600 N /path/to/current/file/being/processed I was able to locate what the ...
  • 4,433
1 vote
2 answers
844 views

How to install vdo/kvdo on Ubuntu 20.04?

I would like to know if there is a way to install Red Hat's vdo in Ubuntu 20.04. So far, I have tried to download the source and compile it, but I get the following error: cc -fPIC -fpic -D_GNU_SOURCE ...
  • 382
6 votes
3 answers
920 views

Finding duplicate files with same filename AND exact same size

I have a huge songs folder with a messy structure and files duplicated in multiple folders. I need a recommendation for a tool or a script that can find and remove duplicates with simple two matches: ...
2 votes
1 answer
222 views

Is there a fuzzy duplicate finder for videos, that does not require a GUI?

I am currently trying to eliminate duplicated videos with minimal changes. Those might be a slightly different encoding, a lower resolution or just changed meta data. These videos are in a complex ...
0 votes
6 answers
234 views

Remove adjacent duplicated words from string

I have a string like this string: one one tow tow three three tow one three How can i remove duplicated words to make it like this: one tow three tow one three The point is that I want to write a ...
0 votes
4 answers
394 views

How to remove duplicate values on the same row using awk?

I want to remove duplicated columns/fields on the same row only. I tried, but I ended up with a long code with nest loops, conditions and arrays that doesn't work correctly. input data: 1 2 3 4 1 2 3 ...
0 votes
2 answers
139 views

Script to find duplicate files by extension and delete them

Recently my NAS was ransomware attacked and all my files were 7zipped. I managed to get the password and extract them and at the same time I renamed the 7zipped file to 7z.bad (so that it's easier ...
2 votes
1 answer
654 views

Is there any advantage to using a hard-link on ZFS instead of relying upon deduplication when considering only disk space allocation?

If I want to create multiple instances of a file on a ZFS file system, is there any advantage to using a hard-link instead of relying upon deduplication as a method of preserving disk space? This ...
  • 2,225
0 votes
0 answers
63 views

I want to write a shell script to delete duplicates in a directory

I do find ./ -type f -exec md5 '{}' \; > tempfile tempfile has lines like this: MD5 (.//Photos-10/IMG_20200901_183050612.jpg) = 2e1f245d195b8d2c3z926dbe0410f7b5 I want to check for repeated ...
2 votes
2 answers
550 views

Deduplicating Files while moving them to XFS

I've got a folder on a non reflink-capable file system (ext4) which I know contains many files with identical blocks in them. I'd like to move/copy that directory to an XFS file system whilst ...
0 votes
1 answer
626 views

How to use `rmlint` to remove duplicates only from one location and leave all else untouched?

I have two locations /path/to/a and /path/to/b. I need to find duplicate files in both paths and remove only items in /path/to/b items. rmlint generates quite a large removal script, but it contains ...
2 votes
1 answer
110 views

FIDEDUPERANGE ioctl doesn't behave as expected on btrfs

According to ioctl_fideduperange, The maximum size of src_length is filesystem dependent and is typically 16 MiB. However, I've been able use src_length of > 1 Gib successfully with a single call ...
  • 694
1 vote
1 answer
287 views

How to use rmlint to merge two large folders?

In exploring options to merge two folders, I've come across a very powerful tool known as rmlint. It has some useful documentation (and Gentle Guide). I have a scenario that I previously mentioned and ...
1 vote
2 answers
406 views

Finding duplicate files using bash script

How do you write a bash one-liner that will find binary files with identical contents, permissions, and owner on the same ext4 file-system, from the current working directory recursively, and replace ...
0 votes
1 answer
103 views

Find duplicate paragraphs in two files and delete one

I have two bib files, some of the entries are duplicates, the duplicates entries are in paragraphs, or could be identified with same pattern, e.g. a.bib looks like @InProceedings{Arranged, author = {...
1 vote
1 answer
21 views

Separately storing parts of text files and their reconstruction: symlinks with multiple targets?

I have two text files whose headers are different, while their contents is the same. $ cat original_file_v1 header 1 beginning header 1 contents header 1 end common contents line 1 common contents ...
  • 4,583
1 vote
1 answer
47 views

Hard link duplicate files based on just size

I'm currently running rdfind on a directory containing more than 4TB of files. Since the checksum part takes an inordinate amount of time I'm looking for alternatives. I know fairly certain that there ...
  • 125
2 votes
0 answers
110 views

dedup seems to be working, but free space says otherwise

I'm testing out dedup with zol on Debian. I've set dedup=on on the zvol before copying any data, then copied data to the mounted volume with no problems. Then I've run cp -r folder1/* folder2/ on test ...
2 votes
0 answers
248 views

Can I share a flatpak directory across hosts?

The context is more or less as follows: I run Distro Foo as my main driver. Inside that I have also set up a few chroots or schroots with Distro Bar, Distro Baz, and an older (or newer) version of ...
0 votes
2 answers
90 views

Remove duplicated first fields of a CSV file

I try to remove repetitions of the same value of the first column from a CSV file without changing the other cell contents and alignment (in other columns). My txt: ACCIDENT EP 4 STEM PERCUS,, ...
3 votes
2 answers
257 views

Remove duplicate lines from files recursively in place but leave one - make lines unique across files

I have many folders and folders contain files. The same line might appear multiple times in single file and/or in multiple files. Files are not sorted. So there are some lines duplicated across ...
  • 131
1 vote
1 answer
616 views

Append text file with command output, but replace the words that already exist (don't add the same text twice)

I am appending a command output to a text file. But if I do it again it will have the same text twice in the text file. Is there a way with for example sed that if a word already exist don't add a new ...
  • 15
0 votes
1 answer
514 views

How to run Shredder Duplicate Finder (rmlint --gui) on Debian? ("Failed to load shredder: No module named 'shredder'")

I'd like to run the rmlint GUI (Shredder) on Debian10 but I get this error: Failed to load shredder: No module named 'shredder' This might be due to a corrupted install; try reinstalling.
  • 3,440
2 votes
0 answers
162 views

Compare disks for missing files with differing directory structure

Apologies for the length of this post, I've tried to keep it short as possible. I'm looking for a tool / method which, given two paths, would show which files are not present in one of the paths. The ...
  • 121
3 votes
0 answers
447 views

Backing up a deduplicated BTRFS filesystem

I have some long-term data in a BTRFS volume. I've been using btrfs-dedupe to deduplicate my data and I'm able to save a tremendous amount of disk space between filesystem compression and ...
  • 36.4k
6 votes
2 answers
2k views

Is there a block-level storage file system?

I'm looking for a file system that stores files by block content, therefore similar files would only take one block. This is for backup purposes. It is, similar to what block-level backup storage ...
  • 165
8 votes
1 answer
4k views

Is there a way to enable reflink on an existing XFS filesystem?

I currently have a 4TB RAID 1 setup on a small, personal Linux server, which is formatted as XFS in LVM. I am interested in enabling the reflink feature of XFS, but I did not do so when I first ...
  • 191
4 votes
1 answer
960 views

How to copy multiple snapshots at once without duplicating data?

I have a live btrfs filesystem of 3.7TiB that's >90% full including old snapshots and a fresh 4TB backup harddisk. How to copy all existing snapshots to the backup harddisk? I tried # btrfs send ...
2 votes
1 answer
49 views

Checking identical files in Linux and deleting according to location

I use fdupes to find and delete identical files. But I want to be able to say something like this ... find all the files that are duplicate in directory A or its subdirectories if there's a ...
  • 977
0 votes
1 answer
77 views

Report directories with contents that exist elsewhere even if scattered

I want to generate a report of directories that I know I can safely delete (even if requiring a quick manual verification), because I know that the full contents all the way down, exist elsewhere--...
  • 299
1 vote
2 answers
3k views

Identify unique records on CSV based on specific columns [duplicate]

If the input is foo,bar,baz bar,baz,qux qux,quux,baz bar,foo,qux waldo,fred,garply the output should be foo,bar,baz bar,baz,qux waldo,fred,garply As you can see, records are deduplicated based on ...
-2 votes
1 answer
259 views

Command to delete duplicate files from current directory [duplicate]

I have a directory with several duplicate files, created by a program. The duplicates have the same name (except for a number), but not all files with the same name are duplicates. What's a simple ...
  • 475
7 votes
1 answer
1k views

What does a rmlint's "clone" for btrfs do?

I was reading the rmlint manual, and one of the duplicate handlers are clone and reflink: · clone: btrfs only. Try to clone both files with the BTRFS_IOC_FILE_EXTENT_SAME ioctl(3p). This will ...
  • 9,132
11 votes
5 answers
28k views

How to find duplicate lines in many large files?

I have ~30k files. Each file contains ~100k lines. A line contains no spaces. The lines within an individual file are sorted and duplicate free. My goal: I want to find all all duplicate lines across ...
4 votes
6 answers
2k views

How to append multiple lines to a file, if these lines doesn't exist in that file?

How to append multiple lines to a file, if these lines doesn't exist in that file? For example, to add multiple global aliases to /etc/bash.bashrc I use an heredocument: cat <<-"BASHRC"...
  • 1,510
1 vote
1 answer
185 views

script for moving kept files deleted with fdupes or other deduplicating package

After I recovered my files with PhotoRec, I ran fdupes $ fdupes -rdNI --sameline /home/user > fdupes.txt. I couldn't use the option -d because round about 1.6 million files in my system have been ...
  • 489
1 vote
1 answer
306 views

How can I tell which copies of a file are COW copies?

I have a ZFS system connected to a macintosh over NFS. The system is nearly full, but I have found a large number of duplicate files with jdupes. I don't know how these copies were made, it's ...
  • 9,132
3 votes
2 answers
7k views

How does tar deal with hardlinked files? [duplicate]

I have a 2.5 TB of data that I want to put in a 2TB hard drive to mail somewhere. It's not hopeless, as a very large fraction of the data consists of duplicate files. I am considering using jdupes ...
  • 9,132
4 votes
4 answers
2k views

cp --reflink=auto for MacOS X

cp --reflink=auto shows following output for MacOS: cp: illegal option -- - Is copy-on-write or deduplication supported for HFS? How can I COW huge files with HFS?
8 votes
1 answer
1k views

Make tar (or other) archive, with data block-aligned like in original files for better block-level deduplication?

How can one generate a tar file, so the contents of tarred files are block-aligned like in the original files, so one could benefit from block-level deduplication ( https://unix.stackexchange.com/a/...