Questions tagged [duplicate]

This tag is ambiguous. Please use a more precise tag instead.

Filter by
Sorted by
Tagged with
0 votes
0 answers
22 views

Linux home dir has a "duplicate" under /home/music but when anything is deleted from one it disappears from both -- how can I delete 2nd home dir?

I don't know where the 2nd "phantom" home dir came from. Properties folder shows it's 4K. The "real" home directory is 128,000+ files and 208GB. User10489 had the answer: a symlink ...
R E Brinson's user avatar
0 votes
3 answers
30 views

Filter duplicated based on values of another column

I have the following example of a dataframe.Where you see that elements of 3nd column could be duplicated.I want to keep the entry which has the highest value in column 5 Meaning that for AGCCCGGGG I ...
Anna Antonatou -Pappaioannou's user avatar
0 votes
3 answers
93 views

Need assistance with awk/sed to identify/mark duplicate IP addresses

Good day. I have a text file which contains pod/node names and associated IPv6 addresses of which two pods have the same IP address, first pod k8-worker0001c-cif-9d86d6dd4-vf9b9 and last pod k8-...
Bjoern's user avatar
  • 31
0 votes
2 answers
73 views

awk is automatically duplicating some lines. Can someone explain?

My data looks like: A 4 G 1 G 1 C 4 C 2 C 2 T 6 T 5 T 5 A 6 T 2 T 2 C 6 T 2 T 2 T 6 G 2 G 2 I am trying the command: awk -F " " '$1==$3 {$7=$6; print $0;} $1==$5 {$7=$4; print $...
user563991's user avatar
1 vote
1 answer
403 views

Find duplicate IPs for different MACs

Using arp-scan to get a list of returned duplicate IP address. However, arp-scan will list duplicate IP with the same MAC address. I get a sorted output of asx.txt (shortened for brevity) arp-scan 172....
JohnGoutbeck's user avatar
0 votes
0 answers
122 views

Removing duplicate values based on two columns

I have a file that would like to filter duplicate values based column 1 and 6 ID,sample,NAME,reference,app_name,appession_id,workflow,execution_status,status,date_created 1,ABC,XYZ,DOP,2022-08-18 13:...
nbn's user avatar
  • 113
1 vote
1 answer
258 views

How to sort a list of strings that contain a combination of letters and numbers

I want to sort following strings by the number and remove duplication in a file cat311 celine434 celine434 celine5 jimmy12 john44 john41 to be celine5 jimmy12 john41 john44 cat311 celine434
user8090410's user avatar
0 votes
0 answers
51 views

Delete duplicated contents from files

I have many backups of a same file. Is there a way to transform that into incremental backup? Those files aren't exactly the same (not same timestamps sometimes, sometimes new data appended here and ...
aac's user avatar
  • 45
0 votes
1 answer
1k views

How do I append the outputs of the echo -e command into two separate files, simultaneously?

First, Thank you for your time and effort to help me with this situation. Now, without further ado, my issue. I am a student in Cybersecurity and I am attempting to make bash script to enumerate my ...
James Smith's user avatar
-1 votes
1 answer
107 views

Skip line from console if equal than line before, adding count (in realtime)

Using uniq it is possible to filter out sequential duplicate lines. while (true) do echo 1; echo 2; echo 2; echo 1; sleep 1; done | uniq becomes: 1 2 1 Is there a way to have duplicated sequential ...
Jose Gómez's user avatar
-1 votes
1 answer
1k views

remove duplicate lines across multiple txt files [closed]

I have 12 text files all in one folder, each with about 5 million lines, each file has no duplicate line on its own but there are duplicated across multiple files, I want to remove the duplicate lines ...
Surprise Awofemi's user avatar
1 vote
2 answers
782 views

Remove duplicates of specific line keeping only the first appearance of each without touching other unspecified duplicates

I'm trying to edit a text file containing several duplicates. The goal is to keep only the first match of a string and remove the rest duplicate lines of the same string. In the example file * Title 1 ...
yeyin33455's user avatar
1 vote
1 answer
1k views

ZSH - PATH Duplication : Directory added at end of PATH keeps duplicating when re-opening Terminal Session

I have recently installed PIPX on MAC running Big Sur and ZSH shell. During the install it prompted for the following to be added to the .zshrc file.... # Created by `pipx` on 2021-03-20 14:22:23 ...
KB_cov's user avatar
  • 19
0 votes
0 answers
148 views

I have a `raspivid` stream, which I'm piping to `ffmpeg`, now i'd like to also stream a raw version of it to a socket?

I have a process outputing an MJPEG video stream, which I pipe into ffmpeg to reduce framerate and then to a socket. raspivid -t 999999 -cd MJPEG -w 1920 -h 1080 -o - | ffmpeg -i - -f mjpeg -r 2 - | ...
Ivan Koshelev's user avatar
1 vote
1 answer
568 views

How to use rmlint to merge two large folders?

In exploring options to merge two folders, I've come across a very powerful tool known as rmlint. It has some useful documentation (and Gentle Guide). I have a scenario that I previously mentioned and ...
ylluminate's user avatar
1 vote
3 answers
455 views

Duplicate Debian dedicated server to a virtual machine

We have a dedicated bare metal debian 10 server which runs some applications which were very hard to setup. Now this server needs to be migrated to a KVM based virtual machine. My plan was instead of ...
manifestor's user avatar
  • 2,463
2 votes
2 answers
995 views

Find duplicate files based on first few characters of filename

I am looking for a way in Linux shell, preferably bash to find duplicates of files based on first few letters of the filenames. Where this would be useful: I build mod packs for Minecraft. As of 1.14....
Kreezxil's user avatar
1 vote
2 answers
2k views

Why ubuntu uses "move to trash" instead of delete?

Does anyone know why in the GUI of Ubuntu(and perhaps the other Linux distributions as well) use the term "move to trash" instead of delete? I want to know what was the historic event that leads to ...
Bat's user avatar
  • 151
0 votes
1 answer
898 views

How to combine multiple text files and remove duplicates effectively?

I tried many things but most of them just got stuck or didn't remove the duplicates properly. It doesnt matter if the output file is sorted or not If file1.txt contains a b c d e c d If file2.txt ...
Aeden Thomas's user avatar
0 votes
2 answers
62 views

Duplicate lines that included same strings after /film/

Explaining is hard I want to start with example. Please see the Bold text. I have very long list like the following list : http://dl3/film/**Daylights.End**.3******************* http://dl45/film/**...
Kaveh202's user avatar
-1 votes
3 answers
781 views

AWK How to take value from header and print it in columns?

I have to get some values from header in txt files and duplicate them as column the number of rows change from file to file , here is an example. Input file: PROJECT : PRK33 X_COR : 258524 Y_COR : ...
user395451's user avatar
0 votes
3 answers
89 views

Duplicate and alter contents of file

I'm wondering if anybody could suggest a way to duplicate the contents of a file, in such a way that the duplicates have been altered? For example, file pre duplication: Nempar|EIJ87098.1 Ecanceri|...
Jahcub Trew's user avatar
0 votes
2 answers
596 views

How to remove duplicate lines inside a text file & get the number of the removed lines?

I know that with awk '!seen[$0]++' filename > output.txt it is possible to remove all duplicates from a text file.But how could I get a list of the removed lines in order to use this information ...
user3776738's user avatar
0 votes
1 answer
111 views

How to find duplicate directory paths even if the contents are different?

I've searched high and low but it seems all but one (Find and list duplicate directories) topic I've found actually deals with my situation, and the result isn't quite what I need. Edit: Here's some ...
user2752794's user avatar
2 votes
1 answer
562 views

Why is a duplicate file sometimes created after editing a file? [duplicate]

I use gedit to modify/create files in my system, and sometimes I see that after editing, a duplicate file is created by the name of samename~ Just that ~ is extra. Why does this happen? Is there any ...
user avatar
2 votes
1 answer
303 views

Check target file for duplicate entry before copy data from source file [duplicate]

I am trying to copy lines from source.txt to target.txt. I would like this bash script to check each line on target.txt if there is duplicate entry before copy. source.txt contains: a$$a$$a b**b**...
danone's user avatar
  • 304
2 votes
1 answer
435 views

check duplicate login from file

I have a question about my code. I'm writing a Bourne shell script that get a user list from server by who command then put it into a file then, it will check the name by read line and if the script ...
Prin Puyakul's user avatar
1 vote
4 answers
3k views

Keeping lines repeated a set number of times [duplicate]

I have a file with multiple columns and have identified lines where values in column 3 have been duplicated. Example Input: A B C 1 2 APPLE 3 4 PEAR 9 3 LEMON 8 3 ORANGE 8 2 APPLE 3 4 APPLE 9 3 ...
Bob's user avatar
  • 349
0 votes
2 answers
1k views

Duplicate and NAT outgoing UDP packets through two internet-facing interfaces.

I have a router, running Debian/Linux 4.1, which has: eth0: 192.168.1.1 attached to a LAN, running DHCPd for wired clients. wan0: 1.2.3.4 attached to the internet wan1: 5.6.7.8 attached to the ...
fadedbee's user avatar
  • 837
0 votes
5 answers
2k views

Number of same line in each text file

I have more then 30 different text files and each one of them has a same word which repeated different time for example in text1 "esr" repeated 12 times and in text2 "esr" repeated 21 times. Is it ...
user avatar
3 votes
3 answers
151 views

How to remove a specific, duplicated line within a file?

I'm looking for a way to remove one specific line from a bunch of files, but only if it occurs more than once in that file. Other lines should be kept, even if they are duplicates. For example, a ...
neXus's user avatar
  • 141
0 votes
3 answers
3k views

How to remove duplicates using awk in 1th~46th column of 47 field csv file?

I have a bash script to combine many files into a single csv file. This csv file has 47 fields and several 10000 rows. To remove duplicates I was using awk '!seen[$0]++' however, for sorting I have ...
triedonce's user avatar
1 vote
1 answer
3k views

How to remove duplicated directories that contain the same files?

I have multiple directories that contain text files. Some of these directories are named differently yet contain the same text files. How do I remove all duplicated directories?
Sean Nolan's user avatar
-1 votes
1 answer
218 views

How to find lines that have the same two fields and remove all but one of those lines [duplicate]

I think it's easier if I break this question into parts. Is there a way to find lines that have the same values in the first field and the same values in the last field? - but not necessarily equal ...
speld_rwong's user avatar
8 votes
7 answers
19k views

How to find and delete duplicate files within the same directory?

I want to find duplicate files, within a directory, and then delete all but one, to reclaim space. How do I achieve this using a shell script? For example: pwd folder Files in it are: log.bkp log ...
Su_scriptingbee's user avatar
16 votes
12 answers
38k views

Remove all duplicate word from string using shell script

I have a string like "aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc" I want to remove duplicate word from string then output will be like "aaa,bbb,ccc" I tried This code Source $ echo "zebra ant spider ...
Urvashi's user avatar
  • 333
0 votes
1 answer
85 views

how to cut & paste and duplicate images

With the help of this forum I have managed to name my images with Exiftool by their unique date and time. My problem is that I have dozens of folders with hundreds of images in each folder combined ...
Smokeyone's user avatar
2 votes
3 answers
1k views

Filter duplicate filenames with tar

I'm using tar like this, tar -cjpvf a.tar "$pattern1" "$pattern2" The problem is that pattern1 and pattern2 sometimes match the same file, but tar doesn't seem to filter these duplicates. e.g The ...
daisy's user avatar
  • 54.4k
1 vote
3 answers
6k views

How to remove duplicate source list on Debian 8.7

Am running Debian 8.7, after installing #letsencrypt following the tutorial on Digital Ocean. I couldn't update my operating system any more, If I run sudo apt-get update it displays the following ...
Ugochukwu Dominic Agbams's user avatar
0 votes
1 answer
79 views

How to delete lines in text file that contain three dupliate fields (strings sep by spaces)

I have a long text file as follows (generated from download links and downloaded filenames): file01 48976 5873987 48976 48976 thdshiesde file02 29876 589845637 29876 29876 uyjedrty 54 file03 38745 ...
speld_rwong's user avatar
3 votes
1 answer
2k views

Remove duplicate lines from multiple files in a folder: Check only files within a range

I had a question about removing duplicate lines in multiple files and was provided with a useful script here: Remove duplicate lines from multiple JSON files while preserving file structure. The ...
Zwentibold's user avatar
0 votes
1 answer
239 views

Dealing with duplicate hash data in text file - keeping one unique line and deleting duplicates

Thanks to this forum I was able to process a large text file containing filenames and hashes and end up with a list of files with identical hashes as shown below: file_35.txt ...
speld_rwong's user avatar
0 votes
2 answers
2k views

Remove duplicate lines from multiple json files while preserving file structure

I have a folder with thousands of json files. The contents of each folder look something like this: file.1424-417982.json file.1424-417995.json file.1424-418013.json file.1424-418015.json (etc.) ...
Zwentibold's user avatar
4 votes
1 answer
756 views

Text file containing filenames and hashes - extracting lines with duplicate hashes

I have generated a large text file containing filenames and sha-256 hashes using the format below - new line at the end of each line after the hashes. file_1.txt ...
speld_rwong's user avatar
3 votes
2 answers
649 views

Deduplicate image files

[user@notebook foobar]$ ls *.jpg|wc -l 1959 [user@notebook foobar]$ cksum * | cut -d' ' -f-2 | sort | uniq -di | wc -l 698 [user@notebook foobar]$ There are many jpg files in a directory. Many of ...
pepite's user avatar
  • 1,083
0 votes
1 answer
333 views

Comparing files in awk/bash

I came across a problem and have no idea how to find the OPTIMAL solution. I have a list of files that looks like this: file1\0file2\0...fileX\0\0file(x+1)\0 Every name of file is separated by a \0 ...
ogarogar's user avatar
  • 191
0 votes
2 answers
178 views

How to duplicate an paragraph with one changed string

I'm looking for something that could copy a paragraph, change the user and insert it in same file. file before: user1 this is only a test of a lovely idea user2 this user shhould be ...
Milky Way's user avatar
4 votes
4 answers
6k views

Finding duplicate users on CentOS

I'm using the following code to find the duplicate username. However, it gives an error. #!/bin/bash cat /etc/passwd | cut -f1 -d":" | /bin/sort -n | /usr/bin/uniq -c |\ while read x; do [ -z "${...
jacqueline fernandez's user avatar
1 vote
1 answer
1k views

Find case duplicate files and rm uppercase

I have a number of image files that have duplicates across case (ie file.jpg File.jpg). I need a script like the following except I want to completely remove all uppercase. I know that fslint can do ...
dras's user avatar
  • 13
5 votes
3 answers
5k views

Find and list duplicate directories

I have directory that has a number of sub-directories and would like to find any duplicates. The folder structure looks something like this: └── Top_Dir └── Level_1_Dir ├── standard_cat ...
dino's user avatar
  • 51