The tag has no usage guidance.

learn more… | top users | synonyms

81
votes
17answers
25k views

Is there an easy way to replace duplicate files with hardlinks?

I'm looking for an easy way (a command or series of commands, probably involving find) to find duplicate files in two directories, and replace the files in one directory with hardlinks of the files in ...
56
votes
5answers
32k views

Find duplicate files

Is it possible to find duplicate files on my disk which are bit to bit identical but have different file-names?
15
votes
5answers
7k views

case-insensitive search of duplicate file-names

I there a way to find all files in a directory with duplicate filenames, regardless of the casing (upper-case and/or lower-case)?
14
votes
5answers
11k views

How can I copy a file and create the target directories at the same time?

I want to cp aaa/deep/sea/blob.psd to bbb/deep/sea/blob.psd How do I do the copy if the deep and sea directories don't exist under bbb so that the copy both creates the directories that are needed ...
9
votes
8answers
649 views

Find duplicate PDF files by content

Some journals generate a different PDF for each download. APS for example stores time and the IP address in the PDF. Or there is a paper version with hyper links and one with text references. How ...
9
votes
1answer
4k views

Open source duplicate image finder for Linux?

Is there a free and open source duplicate image finder for Linux based systems? Finding exact duplicates (based on content, not file name) is sufficient for me, but the ability to find similar images ...
8
votes
5answers
1k views

Remove duplicate lines from a file that contains a timestamp

This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp. Is it possible to tell awk to ...
7
votes
3answers
3k views

Remove duplicate mp3 with different name, size, and hash

I have a massive music library (all mp3), but I some of the music is almost the same but: Maybe one or two second longer About 97% the same as another song Or another bitrate. Is there a way to ...
6
votes
3answers
557 views

Shell script to search files for identical text entries

Here's a nice teaser for a shell scripting guru: Take a directory with several text files. Can be a few up to ~1000. All files contain an identifier on a given line (always the same line). Identify ...
5
votes
4answers
3k views

Use basename to parse a list of paths held in a file

I'm running Mac OSX and trying to use the command line to find the number of files I have with the same name. I tried to use the following command: find ~ -type f -name "*" -print | basename | sort ...
5
votes
4answers
301 views

Removing lines that are common to the outputs produced by two commands

Say, I have two commands that return some text. For example: $ ./c1 /usr/bin/foo /usr/bin/bar /usr/bin/baz $ ./c2 /usr/bin/foo /usr/bin/qux /usr/bin/buzz /usr/bin/bar I want to remove the duplicate ...
4
votes
1answer
242 views

Count and merge consecutive patterns

I'm searching for a short snippet to find, count and merge consecutive duplicates using standard tools or a common scripting language. Say our input is: 1 2 2 2 7 22 a b b c c c c c d dd 2 2 c c ...
4
votes
3answers
2k views

How to remove duplicate files using bash

I have a folder with duplicate (by md5sum (md5 on a Mac)) files, and I want to have a cron job scheduled to remove any found. However, I'm stuck on how to do this. What I have so far: md5 -r * | ...
4
votes
2answers
251 views

Print repeating pattern

I have a marathon file from which I want to print the names of the runners, who have made multiple runs. Something like: M, 2:08:58.0, John Lukas, USA, 1997-07-25, Paris, France F, 2:15:04:0, Abebbe ...
4
votes
2answers
319 views

Keeping first instance of duplicates

I have a file with multiple columns and have identified lines where specific column values (cols 3-6) have been duplicated using a bash script. Example input: A B C D E F G 1 2 T TACA A 3 2 Q 3 4 I ...
4
votes
2answers
530 views

Faster way to rename duplicate files (identified by fdupes) in another directory?

I have a directory full of pdf files of journal articles, most of which are named by their bibtex key. Some time ago I made a backup on an external hard drive, but I haven't kept it up to date and ...
4
votes
2answers
1k views

Finding duplicate files and replace them with symlinks

I'm trying to find a way to check inside a given directory for duplicate files (even with different names) and replace them with symlinks pointing to the first occurrence. I've tried with fdupes but ...
3
votes
1answer
240 views

Unix command equivalent for OS X Finder's Duplicate Exactly

What command can reproduce this function in bash? In Finder select File then press ⇧⌥ to make the command visible (⇧⌥⌘D). In AppleScript it's available as tell application "Finder" to duplicate ...
3
votes
2answers
345 views

Finding duplicate files by name ignoring the case in subdirectories

How is it possible to list duplicate file names on a Linux system ignoring the case including all subdirectories Files should not be compared by their content but only by their names. The output ...
3
votes
1answer
2k views

CentOS 6 interfaces bonding, round-robin instead of active-backup, duplicates frames

Two interfaces, eth0 and eth1 are part of a network bonding bond0 on CentOS 6. All worked well under CentOS 5, but after the upgrade to CentOS 6.6, keeping the same configuration, the network works ...
2
votes
4answers
781 views

Search and Delete duplicate files with different names

I have a large music collection stored on my hard drive; and browsing through it, I found that I have a lot of duplicate files in some album directories. Usually the duplicates exist alongside the ...
2
votes
3answers
2k views

Killing all the process of a command except first process

I am sometimes stuck in a situation where a script/command kept in Cron runs more than once because of some reasons (the first instance is not completed fully, the second instance of the same process ...
2
votes
2answers
143 views

Copy website from RedHat to CentOS

We currently have our website and website's CMS hosted on a dedicated host which runs Red Hat. We want to remove Red Hat and install CentOS instead. My question: is there a straight forward way of ...
2
votes
4answers
53 views

How to delete all duplicate hardlinks to a file?

I've got a directory tree created by rsnapshot, which contains multiple snapshots of the same directory structure with all identical files replaced by hardlinks. I would like to delete all those ...
2
votes
2answers
683 views

Can't mount second hard drive on CentOS 6.6 - Duplicate VG name

Yesterday I add a SSD to my PC configuration and I make on it a fresh installation. At the moment of installation I replace my old HDD and there was only the SSD. When the installation finish I make a ...
2
votes
3answers
714 views

check if all files from a folder are also in another folder

I basically have a directory a with lots of images. Now I want to check if all of these images are in directory b. The point is, that lots of images in b ain't directly in b but in subdirectories. ...
2
votes
3answers
396 views

Find files that have a confirmed duplicate in same directory recursively

Say I have the following directory structure: root |-- dirA |-- file.jpg |-- file-001.jpg <-- dup |-- file2.jpg |-- file3.jpg |-- dirB |-- fileA.jpg |-- fileA_ios....
2
votes
0answers
43 views

how to find duplicate folders inside main folder [closed]

I've moved a lot of files and folders from my old nas to my new smb share. The problem is that someone made entire copies of nearly every folder on that thing. And I need the space. I'm currently ...
2
votes
0answers
38 views

Space efficiency of Btrfs reflinks vs hardlinks

A webpage referenced by the Wikipedia article on Btrfs claims that (emphasis mine) [Btrfs] reflinks have the same use as hardlinks, but are more space efficient. I thought that the opposite was ...
2
votes
0answers
144 views

Removing duplicate files but ignoring certain lines (e.g. embedded timestamp)

I have a bunch of gzipped text files. I am trying to remove duplicate files (using fdupes) but the problem is these files contain a line that has their timestamp in them, and the files are identical ...
1
vote
5answers
84 views

Remove repetitive entries from a file

There is a file with a couple of identical and repetitive entries, which looks something like this: 123 abc nhjk 123 abc cftr 123 abc xdrt 123 def nhjk 123 def cftr 123 def xdrt If the combination ...
1
vote
2answers
716 views

Find duplicate file names with specific matching pattern

I have a folder with some files (snippet of the contents of the folder) PAT1.URGRSVP.50.WR786842JOB11632.WRS20140.FILE0005.DAT PAT1.URGRSVP.50.WR786842JOB11643.WRS20140.FILE0003.DAT PAT1.URGRSVP.51....
1
vote
1answer
253 views

Identifying duplicate fields and print both with awk

I have a file with multiple columns and want to identify those where specific column values (cols 3-6) have been duplicated. The following code finds the duplicates but I want to display both ...
1
vote
2answers
136 views

Duplicate stdin to stdout and stderr, but in a synchronized way

I need to duplicate the stdout of a producer and feed it to two consumer in a synchronized fashion. consumer 1 producer | duplicator | consumer 2 ...
1
vote
1answer
25 views

Find and list duplicate directories

I have directory that has a number of sub-directories and would like to find any duplicates. The folder structure looks something like this: └── Top_Dir └── Level_1_Dir ├── standard_cat ...
1
vote
1answer
241 views

How to safely move a file to another directory and remove duplicates

I want to move files from Downloads, Documents, Pictures, Music, et al to shared locations. This is for various reasons, including because I want to keep media in a central, shared place. I want to ...
0
votes
3answers
1k views

Do I have duplicate packages?

I'm on linux mint 12, when i run sudo apt-get update i see: W: Duplicate sources.list entry http://www.duinsoft.nl/pkg/ debs/all i386 Packages (/var/lib/apt/lists/www.duinsoft....
0
votes
1answer
234 views

Ping a list of IP addresses and outputing duplicates

In Linux how can I ping a list of IP addresses (in a text file) and only output those that have duplicate ping replies?
0
votes
1answer
95 views

Identifying duplicate fields and REMOVE both with awk

Previously, I asked the question: Identifying duplicate fields and print both with awk. I have a file with multiple columns and wanted to identify lines where specific column values (cols 3-6) had ...
0
votes
2answers
50 views

Prints duplicate lines based on first column

I want to get lines where first column duplicates (I used number 32 becouse first column has always 32 characters). So far I've figured out: sort | uniq -d -w 32 filename The problem is that I want ...
0
votes
2answers
31 views

Merge labels from lines with duplicate fields

I have a rather large text file(thousands of lines) in following format: 123.123 5 123.123 91 156.456 45 233.185 85 233.185 4 257.448 1 455.456 60 455.456 2 The first column are ...
0
votes
1answer
166 views

Renaming Duplicate Files with Flatten Folders Command

This command will move the contents of one or more folders to the parent folder. But if there are identical file names with the same extension then there will be files that are overwritten during the ...
0
votes
2answers
597 views

Get or filter duplicated lines by column

I have a file with three columns and I would like to get duplicated lines by third column, for example: AAA = 342 BLABLABLA = 2 BBBx2 = 23 1+1 = 2 KOKOKO= 5 2x1 = 2 The output sould ...
0
votes
1answer
36 views

Keeping first instance of duplicates when column values are interchangeable

I have previously asked a question about keeping the first instance of duplicates and the solution provided has been working perfectly until now (see Keeping first instance of duplicates). However, I ...
0
votes
1answer
420 views

Remove duplicated Java packages

I'm on linux mint 12 I have installed Java, following the NetBeans instructions. It wasn't working on chrome, so I installed it like that: sudo echo 'deb http://www.duinsoft.nl/pkg debs all' >>...