Questions tagged [deduplication]

For questions where a task is to be applied to only one instance of multiple copies of data (files or blocks of data on a filesystem, or strings in a text), or where duplicates of the first such instance are to be ignored for space/time saving purposes.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

find duplicates inside archives

I have a huge archive folder and want to integrate that into a local backup folder and then eliminate duplicates before making a cloud backup of the whole backupfolder. The problem: the files in the ...
user avatar
  • 1
0 votes
2 answers
41 views

Why do these two commands to write text-processing results back to the input file behave so differently?

I have a file authorized_keys and want to deduplicate the content, i.e. remove duplicate entries. I found two possible solutions to achieve this: Use cat and uniq, then redirect the output to the ...
user avatar
  • 1
0 votes
1 answer
38 views

Why did sort -u removed duplicates only in pipe?

In this command, sort -u removed duplicates. curl https://en.wikipedia.org/wiki/Help:Special_page -s | grep -oP 'Special:\K[a-zA-Z0-9]*' | sort -u > special_page_names In this command, it didn't. ...
user avatar
  • 115
1 vote
1 answer
63 views

How to let already copied files share fragments (reflink)?

I copy a file to a different XFS volume on daily basis as follows: # on monday cp --sparse=always /mnt/disk1/huge.file /mnt/disk2/monday/huge.file # on tuesday cp --sparse=always /mnt/disk1/huge.file /...
user avatar
  • 103
0 votes
1 answer
150 views

lvmvdo doesn't deduplicate my data

I am installing lvmvdo con Debian 11.2 to store Proxmox vm disk, doing the following : 1) apt install -y build-essential libdevmapper-dev libz-dev uuid-dev git sudo libblkid-dev man vim dwarves dkms ...
user avatar
1 vote
1 answer
38 views

Meaning of Deduplicated during Borg Create's Realtime Output

When the borg create command is used with the --progress argument, it output like this: 5.50 GB O 5.10 GB C 23.95 kB D 15600 N /path/to/current/file/being/processed I was able to locate what the ...
user avatar
  • 4,311
0 votes
2 answers
531 views

How to install vdo/kvdo on Ubuntu 20.04?

I would like to know if there is a way to install Red Hat's vdo in Ubuntu 20.04. So far, I have tried to download the source and compile it, but I get the following error: cc -fPIC -fpic -D_GNU_SOURCE ...
user avatar
  • 301
6 votes
3 answers
597 views

Finding duplicate files with same filename AND exact same size

I have a huge songs folder with a messy structure and files duplicated in multiple folders. I need a recommendation for a tool or a script that can find and remove duplicates with simple two matches: ...
user avatar
2 votes
1 answer
85 views

Is there a fuzzy duplicate finder for videos, that does not require a GUI?

I am currently trying to eliminate duplicated videos with minimal changes. Those might be a slightly different encoding, a lower resolution or just changed meta data. These videos are in a complex ...
user avatar
0 votes
6 answers
160 views

Remove adjacent duplicated words from string

I have a string like this string: one one tow tow three three tow one three How can i remove duplicated words to make it like this: one tow three tow one three The point is that I want to write a ...
user avatar
0 votes
4 answers
216 views

How to remove duplicate values on the same row using awk?

I want to remove duplicated columns/fields on the same row only. I tried, but I ended up with a long code with nest loops, conditions and arrays that doesn't work correctly. input data: 1 2 3 4 1 2 3 ...
user avatar
0 votes
2 answers
104 views

Script to find duplicate files by extension and delete them

Recently my NAS was ransomware attacked and all my files were 7zipped. I managed to get the password and extract them and at the same time I renamed the 7zipped file to 7z.bad (so that it's easier ...
user avatar
2 votes
1 answer
404 views

Is there any advantage to using a hard-link on ZFS instead of relying upon deduplication when considering only disk space allocation?

If I want to create multiple instances of a file on a ZFS file system, is there any advantage to using a hard-link instead of relying upon deduplication as a method of preserving disk space? This ...
user avatar
  • 2,145
0 votes
0 answers
43 views

I want to write a shell script to delete duplicates in a directory

I do find ./ -type f -exec md5 '{}' \; > tempfile tempfile has lines like this: MD5 (.//Photos-10/IMG_20200901_183050612.jpg) = 2e1f245d195b8d2c3z926dbe0410f7b5 I want to check for repeated ...
user avatar
2 votes
2 answers
327 views

Deduplicating Files while moving them to XFS

I've got a folder on a non reflink-capable file system (ext4) which I know contains many files with identical blocks in them. I'd like to move/copy that directory to an XFS file system whilst ...
user avatar
0 votes
1 answer
401 views

How to use `rmlint` to remove duplicates only from one location and leave all else untouched?

I have two locations /path/to/a and /path/to/b. I need to find duplicate files in both paths and remove only items in /path/to/b items. rmlint generates quite a large removal script, but it contains ...
user avatar
2 votes
1 answer
86 views

FIDEDUPERANGE ioctl doesn't behave as expected on btrfs

According to ioctl_fideduperange, The maximum size of src_length is filesystem dependent and is typically 16 MiB. However, I've been able use src_length of > 1 Gib successfully with a single call ...
user avatar
  • 623
1 vote
1 answer
223 views

How to use rmlint to merge two large folders?

In exploring options to merge two folders, I've come across a very powerful tool known as rmlint. It has some useful documentation (and Gentle Guide). I have a scenario that I previously mentioned and ...
user avatar
1 vote
2 answers
326 views

Finding duplicate files using bash script

How do you write a bash one-liner that will find binary files with identical contents, permissions, and owner on the same ext4 file-system, from the current working directory recursively, and replace ...
user avatar
0 votes
1 answer
71 views

Find duplicate paragraphs in two files and delete one

I have two bib files, some of the entries are duplicates, the duplicates entries are in paragraphs, or could be identified with same pattern, e.g. a.bib looks like @InProceedings{Arranged, author = {...
user avatar
1 vote
1 answer
20 views

Separately storing parts of text files and their reconstruction: symlinks with multiple targets?

I have two text files whose headers are different, while their contents is the same. $ cat original_file_v1 header 1 beginning header 1 contents header 1 end common contents line 1 common contents ...
user avatar
  • 4,493
1 vote
1 answer
42 views

Hard link duplicate files based on just size

I'm currently running rdfind on a directory containing more than 4TB of files. Since the checksum part takes an inordinate amount of time I'm looking for alternatives. I know fairly certain that there ...
user avatar
  • 125
2 votes
0 answers
107 views

dedup seems to be working, but free space says otherwise

I'm testing out dedup with zol on Debian. I've set dedup=on on the zvol before copying any data, then copied data to the mounted volume with no problems. Then I've run cp -r folder1/* folder2/ on test ...
user avatar
2 votes
0 answers
206 views

Can I share a flatpak directory across hosts?

The context is more or less as follows: I run Distro Foo as my main driver. Inside that I have also set up a few chroots or schroots with Distro Bar, Distro Baz, and an older (or newer) version of ...
user avatar
0 votes
2 answers
88 views

Remove duplicated first fields of a CSV file

I try to remove repetitions of the same value of the first column from a CSV file without changing the other cell contents and alignment (in other columns). My txt: ACCIDENT EP 4 STEM PERCUS,, ...
user avatar
3 votes
2 answers
192 views

Remove duplicate lines from files recursively in place but leave one - make lines unique across files

I have many folders and folders contain files. The same line might appear multiple times in single file and/or in multiple files. Files are not sorted. So there are some lines duplicated across ...
user avatar
  • 131
1 vote
1 answer
408 views

Append text file with command output, but replace the words that already exist (don't add the same text twice)

I am appending a command output to a text file. But if I do it again it will have the same text twice in the text file. Is there a way with for example sed that if a word already exist don't add a new ...
user avatar
  • 15
0 votes
1 answer
443 views

How to run Shredder Duplicate Finder (rmlint --gui) on Debian? ("Failed to load shredder: No module named 'shredder'")

I'd like to run the rmlint GUI (Shredder) on Debian10 but I get this error: Failed to load shredder: No module named 'shredder' This might be due to a corrupted install; try reinstalling.
user avatar
  • 3,430
2 votes
0 answers
145 views

Compare disks for missing files with differing directory structure

Apologies for the length of this post, I've tried to keep it short as possible. I'm looking for a tool / method which, given two paths, would show which files are not present in one of the paths. The ...
user avatar
  • 121
3 votes
0 answers
389 views

Backing up a deduplicated BTRFS filesystem

I have some long-term data in a BTRFS volume. I've been using btrfs-dedupe to deduplicate my data and I'm able to save a tremendous amount of disk space between filesystem compression and ...
user avatar
  • 35.5k
6 votes
2 answers
2k views

Is there a block-level storage file system?

I'm looking for a file system that stores files by block content, therefore similar files would only take one block. This is for backup purposes. It is, similar to what block-level backup storage ...
user avatar
  • 165
7 votes
1 answer
3k views

Is there a way to enable reflink on an existing XFS filesystem?

I currently have a 4TB RAID 1 setup on a small, personal Linux server, which is formatted as XFS in LVM. I am interested in enabling the reflink feature of XFS, but I did not do so when I first ...
user avatar
  • 171
4 votes
1 answer
892 views

How to copy multiple snapshots at once without duplicating data?

I have a live btrfs filesystem of 3.7TiB that's >90% full including old snapshots and a fresh 4TB backup harddisk. How to copy all existing snapshots to the backup harddisk? I tried # btrfs send ...
user avatar
1 vote
1 answer
43 views

Checking identical files in Linux and deleting according to location

I use fdupes to find and delete identical files. But I want to be able to say something like this ... find all the files that are duplicate in directory A or its subdirectories if there's a ...
user avatar
  • 937
0 votes
1 answer
66 views

Report directories with contents that exist elsewhere even if scattered

I want to generate a report of directories that I know I can safely delete (even if requiring a quick manual verification), because I know that the full contents all the way down, exist elsewhere--...
user avatar
  • 291
1 vote
2 answers
3k views

Identify unique records on CSV based on specific columns [duplicate]

If the input is foo,bar,baz bar,baz,qux qux,quux,baz bar,foo,qux waldo,fred,garply the output should be foo,bar,baz bar,baz,qux waldo,fred,garply As you can see, records are deduplicated based on ...
user avatar
-2 votes
1 answer
207 views

Command to delete duplicate files from current directory [duplicate]

I have a directory with several duplicate files, created by a program. The duplicates have the same name (except for a number), but not all files with the same name are duplicates. What's a simple ...
user avatar
  • 473
6 votes
1 answer
1k views

What does a rmlint's "clone" for btrfs do?

I was reading the rmlint manual, and one of the duplicate handlers are clone and reflink: · clone: btrfs only. Try to clone both files with the BTRFS_IOC_FILE_EXTENT_SAME ioctl(3p). This will ...
user avatar
  • 9,032
10 votes
5 answers
26k views

How to find duplicate lines in many large files?

I have ~30k files. Each file contains ~100k lines. A line contains no spaces. The lines within an individual file are sorted and duplicate free. My goal: I want to find all all duplicate lines across ...
user avatar
4 votes
6 answers
2k views

How to append multiple lines to a file, if these lines doesn't exist in that file?

How to append multiple lines to a file, if these lines doesn't exist in that file? For example, to add multiple global aliases to /etc/bash.bashrc I use an heredocument: cat <<-"BASHRC"...
user avatar
  • 1,460
1 vote
1 answer
182 views

script for moving kept files deleted with fdupes or other deduplicating package

After I recovered my files with PhotoRec, I ran fdupes $ fdupes -rdNI --sameline /home/user > fdupes.txt. I couldn't use the option -d because round about 1.6 million files in my system have been ...
user avatar
  • 449
1 vote
1 answer
278 views

How can I tell which copies of a file are COW copies?

I have a ZFS system connected to a macintosh over NFS. The system is nearly full, but I have found a large number of duplicate files with jdupes. I don't know how these copies were made, it's ...
user avatar
  • 9,032
3 votes
2 answers
6k views

How does tar deal with hardlinked files? [duplicate]

I have a 2.5 TB of data that I want to put in a 2TB hard drive to mail somewhere. It's not hopeless, as a very large fraction of the data consists of duplicate files. I am considering using jdupes ...
user avatar
  • 9,032
4 votes
4 answers
2k views

cp --reflink=auto for MacOS X

cp --reflink=auto shows following output for MacOS: cp: illegal option -- - Is copy-on-write or deduplication supported for HFS? How can I COW huge files with HFS?
user avatar
8 votes
1 answer
1k views

Make tar (or other) archive, with data block-aligned like in original files for better block-level deduplication?

How can one generate a tar file, so the contents of tarred files are block-aligned like in the original files, so one could benefit from block-level deduplication ( https://unix.stackexchange.com/a/...
user avatar
1 vote
1 answer
56 views

Mountable deduplicating container

I want a container that I can mount which dedups files within it. Basically, I'd like to have a simple backup solution where I mount it, copy my home folder with the name "Home (Feb 9)" and it only ...
user avatar
  • 11
16 votes
1 answer
4k views

How to duplicate a file without copying its data with btrfs?

I have no experience with btrfs, but it's advertised to be able to de-duplicate files. In my application, I'd need to duplicate whole directory trees. From what I learned, btrfs only de-duplicates ...
user avatar
  • 1,083
3 votes
2 answers
111 views

Find files recursively, but choose largest from among those with duplicate names

Given a nested directory structure containing various files, I would like to find all the files within it, but where there are multiple files with the same name, I'd like to return just the largest ...
user avatar
  • 333
2 votes
2 answers
155 views

transfer many similar files over ssh

A service on a linux server is only able to do full backups, where each backup is a .tar archive (no compression). Many contents of the archive do not change from day to day. Each .tar file size is ...
user avatar
  • 2,398
5 votes
1 answer
869 views

Are tars deduplicatable at the block level?

Quite simply, when a tar file is made on disk, would the extents be deduplicatable with extents inside and/or outside of the tar? I am asking in the theoretical sense, so if the extents of data are ...
user avatar
  • 341