Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [duplicate]

This tag is ambiguous. Please use a more precise tag instead.

0
votes
1answer
39 views

How to find duplicate directory paths even if the contents are different?

I've searched high and low but it seems all but one (Find and list duplicate directories) topic I've found actually deals with my situation, and the result isn't quite what I need. Edit: Here's some ...
0
votes
0answers
9 views

How can I fix duplicate fonts after I ran fc-cache?

I messed up. I ran some commands that I didn't understand, and now my font list contains several duplicate entries. sudo dpkg-reconfigure fontconfig-config followed by Enter, Enter, ↑, Enter (...
2
votes
1answer
406 views

Why is a duplicate file sometimes created after editing a file? [duplicate]

I use gedit to modify/create files in my system, and sometimes I see that after editing, a duplicate file is created by the name of samename~ Just that ~ is extra. Why does this happen? Is there any ...
2
votes
1answer
136 views

Check target file for duplicate entry before copy data from source file [duplicate]

I am trying to copy lines from source.txt to target.txt. I would like this bash script to check each line on target.txt if there is duplicate entry before copy. source.txt contains: a$$a$$a b**b**...
2
votes
1answer
159 views

check duplicate login from file

I have a question about my code. I'm writing a Bourne shell script that get a user list from server by who command then put it into a file then, it will check the name by read line and if the script ...
1
vote
4answers
652 views

Keeping lines repeated a set number of times [duplicate]

I have a file with multiple columns and have identified lines where values in column 3 have been duplicated. Example Input: A B C 1 2 APPLE 3 4 PEAR 9 3 LEMON 8 3 ORANGE 8 2 APPLE 3 4 APPLE 9 3 ...
0
votes
2answers
422 views

Duplicate and NAT outgoing UDP packets through two internet-facing interfaces.

I have a router, running Debian/Linux 4.1, which has: eth0: 192.168.1.1 attached to a LAN, running DHCPd for wired clients. wan0: 1.2.3.4 attached to the internet wan1: 5.6.7.8 attached to the ...
1
vote
5answers
763 views

Number of same line in each text file

I have more then 30 different text files and each one of them has a same word which repeated different time for example in text1 "esr" repeated 12 times and in text2 "esr" repeated 21 times. Is it ...
3
votes
3answers
97 views

How to remove a specific, duplicated line within a file?

I'm looking for a way to remove one specific line from a bunch of files, but only if it occurs more than once in that file. Other lines should be kept, even if they are duplicates. For example, a ...
0
votes
3answers
716 views

How to remove duplicates using awk in 1th~46th column of 47 field csv file?

I have a bash script to combine many files into a single csv file. This csv file has 47 fields and several 10000 rows. To remove duplicates I was using awk '!seen[$0]++' however, for sorting I have ...
1
vote
2answers
1k views

How to remove duplicated directories that contain the same files?

I have multiple directories that contain text files. Some of these directories are named differently yet contain the same text files. How do I remove all duplicated directories?
-1
votes
1answer
63 views

How to find lines that have the same two fields and remove all but one of those lines [duplicate]

I think it's easier if I break this question into parts. Is there a way to find lines that have the same values in the first field and the same values in the last field? - but not necessarily equal ...
5
votes
7answers
9k views

How to find and delete duplicate files within the same directory?

I want to find duplicate files, within a directory, and then delete all but one, to reclaim space. How do I achieve this using a shell script? For example: pwd folder Files in it are: log.bkp log ...
10
votes
10answers
9k views

Remove all duplicate word from string using shell script

I have a string like "aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc" I want to remove duplicate word from string then output will be like "aaa,bbb,ccc" I tried This code Source $ echo "zebra ant spider ...
0
votes
1answer
62 views

how to cut & paste and duplicate images

With the help of this forum I have managed to name my images with Exiftool by their unique date and time. My problem is that I have dozens of folders with hundreds of images in each folder combined ...
2
votes
3answers
153 views

Filter duplicate filenames with tar

I'm using tar like this, tar -cjpvf a.tar "$pattern1" "$pattern2" The problem is that pattern1 and pattern2 sometimes match the same file, but tar doesn't seem to filter these duplicates. e.g The ...
1
vote
2answers
3k views

How to remove duplicate source list on Debian 8.7

Am running Debian 8.7, after installing #letsencrypt following the tutorial on Digital Ocean. I couldn't update my operating system any more, If I run sudo apt-get update it displays the following ...
0
votes
1answer
39 views

How to delete lines in text file that contain three dupliate fields (strings sep by spaces)

I have a long text file as follows (generated from download links and downloaded filenames): file01 48976 5873987 48976 48976 thdshiesde file02 29876 589845637 29876 29876 uyjedrty 54 file03 38745 ...
3
votes
1answer
492 views

Remove duplicate lines from multiple files in a folder: Check only files within a range

I had a question about removing duplicate lines in multiple files and was provided with a useful script here: Remove duplicate lines from multiple JSON files while preserving file structure. The ...
0
votes
1answer
59 views

Dealing with duplicate hash data in text file - keeping one unique line and deleting duplicates

Thanks to this forum I was able to process a large text file containing filenames and hashes and end up with a list of files with identical hashes as shown below: file_35.txt ...
0
votes
2answers
779 views

Remove duplicate lines from multiple json files while preserving file structure

I have a folder with thousands of json files. The contents of each folder look something like this: file.1424-417982.json file.1424-417995.json file.1424-418013.json file.1424-418015.json (etc.) ...
4
votes
1answer
413 views

Text file containing filenames and hashes - extracting lines with duplicate hashes

I have generated a large text file containing filenames and sha-256 hashes using the format below - new line at the end of each line after the hashes. file_1.txt ...
1
vote
2answers
293 views

Deduplicate image files

[user@notebook foobar]$ ls *.jpg|wc -l 1959 [user@notebook foobar]$ cksum * | cut -d' ' -f-2 | sort | uniq -di | wc -l 698 [user@notebook foobar]$ There are many jpg files in a directory. Many of ...
0
votes
1answer
159 views

Comparing files in awk/bash

I came across a problem and have no idea how to find the OPTIMAL solution. I have a list of files that looks like this: file1\0file2\0...fileX\0\0file(x+1)\0 Every name of file is separated by a \0 ...
0
votes
2answers
85 views

How to duplicate an paragraph with one changed string

I'm looking for something that could copy a paragraph, change the user and insert it in same file. file before: user1 this is only a test of a lovely idea user2 this user shhould be ...
4
votes
4answers
3k views

Finding duplicate users on CentOS

I'm using the following code to find the duplicate username. However, it gives an error. #!/bin/bash cat /etc/passwd | cut -f1 -d":" | /bin/sort -n | /usr/bin/uniq -c |\ while read x; do [ -z "${...
1
vote
1answer
488 views

Find case duplicate files and rm uppercase

I have a number of image files that have duplicates across case (ie file.jpg File.jpg). I need a script like the following except I want to completely remove all uppercase. I know that fslint can do ...
4
votes
1answer
1k views

Find and list duplicate directories

I have directory that has a number of sub-directories and would like to find any duplicates. The folder structure looks something like this: └── Top_Dir └── Level_1_Dir ├── standard_cat ...
4
votes
4answers
3k views

How to delete all duplicate hardlinks to a file?

I've got a directory tree created by rsnapshot, which contains multiple snapshots of the same directory structure with all identical files replaced by hardlinks. I would like to delete all those ...
3
votes
0answers
910 views

how to find duplicate folders inside main folder [closed]

I've moved a lot of files and folders from my old nas to my new smb share. The problem is that someone made entire copies of nearly every folder on that thing. And I need the space. I'm currently ...
3
votes
0answers
735 views

Space efficiency of Btrfs reflinks vs hardlinks

A webpage referenced by the Wikipedia article on Btrfs claims that (emphasis mine) [Btrfs] reflinks have the same use as hardlinks, but are more space efficient. I thought that the opposite was ...
0
votes
1answer
1k views

Identifying duplicate fields and REMOVE both with awk

Previously, I asked the question: Identifying duplicate fields and print both with awk. I have a file with multiple columns and wanted to identify lines where specific column values (cols 3-6) had ...
1
vote
2answers
920 views

Prints duplicate lines based on first column

I want to get lines where first column duplicates (I used number 32 becouse first column has always 32 characters). So far I've figured out: sort | uniq -d -w 32 filename The problem is that I want ...
1
vote
5answers
119 views

Remove repetitive entries from a file

There is a file with a couple of identical and repetitive entries, which looks something like this: 123 abc nhjk 123 abc cftr 123 abc xdrt 123 def nhjk 123 def cftr 123 def xdrt If the combination ...
0
votes
1answer
117 views

Keeping first instance of duplicates when column values are interchangeable

I have previously asked a question about keeping the first instance of duplicates and the solution provided has been working perfectly until now (see Keeping first instance of duplicates). However, I ...
5
votes
2answers
5k views

Keeping first instance of duplicates

I have a file with multiple columns and have identified lines where specific column values (cols 3-6) have been duplicated using a bash script. Example input: A B C D E F G 1 2 T TACA A 3 2 Q 3 4 I ...
0
votes
2answers
49 views

Merge labels from lines with duplicate fields

I have a rather large text file(thousands of lines) in following format: 123.123 5 123.123 91 156.456 45 233.185 85 233.185 4 257.448 1 455.456 60 455.456 2 The first column are sorted ...
2
votes
2answers
865 views

Duplicate stdin to stdout and stderr, but in a synchronized way

I need to duplicate the stdout of a producer and feed it to two consumer in a synchronized fashion. consumer 1 producer | duplicator | consumer 2 ...
2
votes
1answer
11k views

Identifying duplicate fields and print both with awk

I have a file with multiple columns and want to identify those where specific column values (cols 3-6) have been duplicated. The following code finds the duplicates but I want to display both ...
0
votes
1answer
830 views

Renaming Duplicate Files with Flatten Folders Command

This command will move the contents of one or more folders to the parent folder. But if there are identical file names with the same extension then there will be files that are overwritten during the ...
3
votes
3answers
11k views

Get or filter duplicated lines by column

I have a file with three columns and I would like to get duplicated lines by third column, for example: AAA = 342 BLABLABLA = 2 BBBx2 = 23 1+1 = 2 KOKOKO= 5 2x1 = 2 The output sould be:...
3
votes
1answer
572 views

Unix command equivalent for OS X Finder's Duplicate Exactly

What command can reproduce this function in bash? In Finder select File then press ⇧⌥ to make the command visible (⇧⌥⌘D). In AppleScript it's available as tell application "Finder" to duplicate ...
3
votes
2answers
2k views

Can't mount second hard drive on CentOS 6.6 - Duplicate VG name

Yesterday I add a SSD to my PC configuration and I make on it a fresh installation. At the moment of installation I replace my old HDD and there was only the SSD. When the installation finish I make a ...
7
votes
4answers
6k views

Search and Delete duplicate files with different names

I have a large music collection stored on my hard drive; and browsing through it, I found that I have a lot of duplicate files in some album directories. Usually the duplicates exist alongside the ...
6
votes
3answers
10k views

How to remove duplicate files using bash

I have a folder with duplicate (by md5sum (md5 on a Mac)) files, and I want to have a cron job scheduled to remove any found. However, I'm stuck on how to do this. What I have so far: md5 -r * | ...
3
votes
1answer
6k views

CentOS 6 interfaces bonding, round-robin instead of active-backup, duplicates frames

Two interfaces, eth0 and eth1 are part of a network bonding bond0 on CentOS 6. All worked well under CentOS 5, but after the upgrade to CentOS 6.6, keeping the same configuration, the network works ...
9
votes
8answers
2k views

Find duplicate PDF files by content

Some journals generate a different PDF for each download. APS for example stores time and the IP address in the PDF. Or there is a paper version with hyper links and one with text references. How ...
6
votes
3answers
5k views

check if all files from a folder are also in another folder

I basically have a directory a with lots of images. Now I want to check if all of these images are in directory b. The point is, that lots of images in b ain't directly in b but in subdirectories. ...
4
votes
2answers
694 views

Print repeating pattern

I have a marathon file from which I want to print the names of the runners, who have made multiple runs. Something like: M, 2:08:58.0, John Lukas, USA, 1997-07-25, Paris, France F, 2:15:04:0, Abebbe ...
3
votes
3answers
1k views

Find files that have a confirmed duplicate in same directory recursively

Say I have the following directory structure: root |-- dirA |-- file.jpg |-- file-001.jpg <-- dup |-- file2.jpg |-- file3.jpg |-- dirB |-- fileA.jpg |-- fileA_ios....