The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
1answer
60 views

Finding duplicate files and replace them with symlinks

I'm trying to find a way to check inside a given directory for duplicate files (even with different names) and replace them with symlinks pointing to the first occurrence. I've tried with fdupes but ...
0
votes
0answers
10 views

How to find duplicate files with identical content and identical or different names [duplicate]

I have several directories containing a few hundred files in each. Each directory contains files that are duplicated (i.e. have the same content) within the same directory, and/or are duplicated in ...
2
votes
0answers
67 views

Removing duplicate files but ignoring certain lines (e.g. embedded timestamp)

I have a bunch of gzipped text files. I am trying to remove duplicate files (using fdupes) but the problem is these files contain a line that has their timestamp in them, and the files are identical ...
4
votes
4answers
228 views

Removing lines that are common to the outputs produced by two commands

Say, I have two commands that return some text. For example: $ ./c1 /usr/bin/foo /usr/bin/bar /usr/bin/baz $ ./c2 /usr/bin/foo /usr/bin/qux /usr/bin/buzz /usr/bin/bar I want to remove the duplicate ...
6
votes
3answers
248 views

Shell script to search files for identical text entries

Here's a nice teaser for a shell scripting guru: Take a directory with several text files. Can be a few up to ~1000. All files contain an identifier on a given line (always the same line). Identify ...
4
votes
2answers
189 views

Faster way to rename duplicate files (identified by fdupes) in another directory?

I have a directory full of pdf files of journal articles, most of which are named by their bibtex key. Some time ago I made a backup on an external hard drive, but I haven't kept it up to date and ...
3
votes
4answers
443 views

Use basename to parse a list of paths held in a file

I'm running Mac OSX and trying to use the command line to find the number of files I have with the same name. I tried to use the following command: find ~ -type f -name "*" -print | basename | sort ...
1
vote
2answers
332 views

Find duplicate file names with specific matching pattern

I have a folder with some files (snippet of the contents of the folder) PAT1.URGRSVP.50.WR786842JOB11632.WRS20140.FILE0005.DAT PAT1.URGRSVP.50.WR786842JOB11643.WRS20140.FILE0003.DAT ...
3
votes
1answer
64 views

Count and merge consecutive patterns

I'm searching for a short snippet to find, count and merge consecutive duplicates using standard tools or a common scripting language. Say our input is: 1 2 2 2 7 22 a b b c c c c c d dd 2 2 c c ...
1
vote
1answer
138 views

How to safely move a file to another directory and remove duplicates

I want to move files from Downloads, Documents, Pictures, Music, et al to shared locations. This is for various reasons, including because I want to keep media in a central, shared place. I want to ...
0
votes
1answer
152 views

Ping a list of IP addresses and outputing duplicates

In Linux how can I ping a list of IP addresses (in a text file) and only output those that have duplicate ping replies?
2
votes
2answers
112 views

Copy website from RedHat to CentOS

We currently have our website and website's CMS hosted on a dedicated host which runs Red Hat. We want to remove Red Hat and install CentOS instead. My question: is there a straight forward way of ...
3
votes
2answers
829 views

Remove duplicate mp3 with different name, size, and hash

I have a massive music library (all mp3), but I some of the music is almost the same but: Maybe one or two second longer About 97% the same as another song Or another bitrate. Is there a way to ...
1
vote
3answers
710 views

Killing all the process of a command except first process

I am sometimes stuck in a situation where a script/command kept in Cron runs more than once because of some reasons (the first instance is not completed fully, the second instance of the same process ...
12
votes
5answers
2k views

How can I copy a file and create the target directories at the same time?

I want to cp aaa/deep/sea/blob.psd to bbb/deep/sea/blob.psd How do I do the copy if the deep and sea directories don't exist under bbb so that the copy both creates the directories that are needed ...
0
votes
2answers
706 views

Do I have duplicate packages?

I'm on linux mint 12, when i run sudo apt-get update i see: W: Duplicate sources.list entry http://www.duinsoft.nl/pkg/ debs/all i386 Packages ...
0
votes
0answers
327 views

Remove duplicated Java packages

I'm on linux mint 12 I have installed Java, following the NetBeans instructions. It wasn't working on chrome, so I installed it like that: sudo echo 'deb http://www.duinsoft.nl/pkg debs all' ...
9
votes
1answer
3k views

Open source duplicate image finder for Linux?

Is there a free and open source duplicate image finder for Linux based systems? Finding exact duplicates (based on content, not file name) is sufficient for me, but the ability to find similar images ...
11
votes
5answers
4k views

case-insensitive search of duplicate file-names

I there a way to find all files in a directory with duplicate filenames, regardless of the casing (upper-case and/or lower-case)?
50
votes
15answers
14k views

Is there an easy way to replace duplicate files with hardlinks?

I'm looking for an easy way (a command or series of commands, probably involving find) to find duplicate files in two directories, and replace the files in one directory with hardlinks of the files in ...