2022 Developer Survey is open! Take survey.

Questions tagged [duplicate]

This tag is ambiguous. Please use a more precise tag instead.

Filter by
Sorted by
Tagged with
0 votes
0 answers
29 views

Delete duplicated contents from files

I have many backups of a same file. Is there a way to transform that into incremental backup? Those files aren't exactly the same (not same timestamps sometimes, sometimes new data appended here and ...
user avatar
  • 13
0 votes
1 answer
38 views

How do I append the outputs of the echo -e command into two separate files, simultaneously?

First, Thank you for your time and effort to help me with this situation. Now, without further ado, my issue. I am a student in Cybersecurity and I am attempting to make bash script to enumerate my ...
user avatar
-1 votes
1 answer
31 views

Skip line from console if equal than line before, adding count (in realtime)

Using uniq it is possible to filter out sequential duplicate lines. while (true) do echo 1; echo 2; echo 2; echo 1; sleep 1; done | uniq becomes: 1 2 1 Is there a way to have duplicated sequential ...
user avatar
-1 votes
1 answer
195 views

remove duplicate lines across multiple txt files [closed]

I have 12 text files all in one folder, each with about 5 million lines, each file has no duplicate line on its own but there are duplicated across multiple files, I want to remove the duplicate lines ...
user avatar
1 vote
2 answers
81 views

Remove duplicates of specific line keeping only the first appearance of each without touching other unspecified duplicates

I'm trying to edit a text file containing several duplicates. The goal is to keep only the first match of a string and remove the rest duplicate lines of the same string. In the example file * Title 1 ...
user avatar
0 votes
1 answer
354 views

ZSH - PATH Duplication : Directory added at end of PATH keeps duplicating when re-opening Terminal Session

I have recently installed PIPX on MAC running Big Sur and ZSH shell. During the install it prompted for the following to be added to the .zshrc file.... # Created by `pipx` on 2021-03-20 14:22:23 ...
user avatar
  • 9
0 votes
0 answers
100 views

I have a `raspivid` stream, which I'm piping to `ffmpeg`, now i'd like to also stream a raw version of it to a socket?

I have a process outputing an MJPEG video stream, which I pipe into ffmpeg to reduce framerate and then to a socket. raspivid -t 999999 -cd MJPEG -w 1920 -h 1080 -o - | ffmpeg -i - -f mjpeg -r 2 - | ...
user avatar
1 vote
1 answer
224 views

How to use rmlint to merge two large folders?

In exploring options to merge two folders, I've come across a very powerful tool known as rmlint. It has some useful documentation (and Gentle Guide). I have a scenario that I previously mentioned and ...
user avatar
1 vote
3 answers
264 views

Duplicate Debian dedicated server to a virtual machine

We have a dedicated bare metal debian 10 server which runs some applications which were very hard to setup. Now this server needs to be migrated to a KVM based virtual machine. My plan was instead of ...
user avatar
  • 2,281
1 vote
2 answers
416 views

Find duplicate files based on first few characters of filename

I am looking for a way in Linux shell, preferably bash to find duplicates of files based on first few letters of the filenames. Where this would be useful: I build mod packs for Minecraft. As of 1.14....
user avatar
1 vote
2 answers
692 views

Why ubuntu uses "move to trash" instead of delete?

Does anyone know why in the GUI of Ubuntu(and perhaps the other Linux distributions as well) use the term "move to trash" instead of delete? I want to know what was the historic event that leads to ...
user avatar
  • 151
0 votes
1 answer
277 views

How to combine multiple text files and remove duplicates effectively?

I tried many things but most of them just got stuck or didn't remove the duplicates properly. It doesnt matter if the output file is sorted or not If file1.txt contains a b c d e c d If file2.txt ...
user avatar
0 votes
2 answers
44 views

Duplicate lines that included same strings after /film/

Explaining is hard I want to start with example. Please see the Bold text. I have very long list like the following list : http://dl3/film/**Daylights.End**.3******************* http://dl45/film/**...
user avatar
-1 votes
3 answers
495 views

AWK How to take value from header and print it in columns?

I have to get some values from header in txt files and duplicate them as column the number of rows change from file to file , here is an example. Input file: PROJECT : PRK33 X_COR : 258524 Y_COR : ...
user avatar
0 votes
3 answers
47 views

Duplicate and alter contents of file

I'm wondering if anybody could suggest a way to duplicate the contents of a file, in such a way that the duplicates have been altered? For example, file pre duplication: Nempar|EIJ87098.1 Ecanceri|...
user avatar
0 votes
2 answers
382 views

How to remove duplicate lines inside a text file & get the number of the removed lines?

I know that with awk '!seen[$0]++' filename > output.txt it is possible to remove all duplicates from a text file.But how could I get a list of the removed lines in order to use this information ...
user avatar
0 votes
1 answer
52 views

How to find duplicate directory paths even if the contents are different?

I've searched high and low but it seems all but one (Find and list duplicate directories) topic I've found actually deals with my situation, and the result isn't quite what I need. Edit: Here's some ...
user avatar
2 votes
1 answer
474 views

Why is a duplicate file sometimes created after editing a file? [duplicate]

I use gedit to modify/create files in my system, and sometimes I see that after editing, a duplicate file is created by the name of samename~ Just that ~ is extra. Why does this happen? Is there any ...
user avatar
2 votes
1 answer
221 views

Check target file for duplicate entry before copy data from source file [duplicate]

I am trying to copy lines from source.txt to target.txt. I would like this bash script to check each line on target.txt if there is duplicate entry before copy. source.txt contains: a$$a$$a b**b**...
user avatar
  • 304
2 votes
1 answer
372 views

check duplicate login from file

I have a question about my code. I'm writing a Bourne shell script that get a user list from server by who command then put it into a file then, it will check the name by read line and if the script ...
user avatar
1 vote
4 answers
2k views

Keeping lines repeated a set number of times [duplicate]

I have a file with multiple columns and have identified lines where values in column 3 have been duplicated. Example Input: A B C 1 2 APPLE 3 4 PEAR 9 3 LEMON 8 3 ORANGE 8 2 APPLE 3 4 APPLE 9 3 ...
user avatar
  • 349
0 votes
2 answers
966 views

Duplicate and NAT outgoing UDP packets through two internet-facing interfaces.

I have a router, running Debian/Linux 4.1, which has: eth0: 192.168.1.1 attached to a LAN, running DHCPd for wired clients. wan0: 1.2.3.4 attached to the internet wan1: 5.6.7.8 attached to the ...
user avatar
  • 579
0 votes
5 answers
2k views

Number of same line in each text file

I have more then 30 different text files and each one of them has a same word which repeated different time for example in text1 "esr" repeated 12 times and in text2 "esr" repeated 21 times. Is it ...
user avatar
3 votes
3 answers
127 views

How to remove a specific, duplicated line within a file?

I'm looking for a way to remove one specific line from a bunch of files, but only if it occurs more than once in that file. Other lines should be kept, even if they are duplicates. For example, a ...
user avatar
  • 141
0 votes
3 answers
3k views

How to remove duplicates using awk in 1th~46th column of 47 field csv file?

I have a bash script to combine many files into a single csv file. This csv file has 47 fields and several 10000 rows. To remove duplicates I was using awk '!seen[$0]++' however, for sorting I have ...
user avatar
1 vote
1 answer
3k views

How to remove duplicated directories that contain the same files?

I have multiple directories that contain text files. Some of these directories are named differently yet contain the same text files. How do I remove all duplicated directories?
user avatar
-1 votes
1 answer
159 views

How to find lines that have the same two fields and remove all but one of those lines [duplicate]

I think it's easier if I break this question into parts. Is there a way to find lines that have the same values in the first field and the same values in the last field? - but not necessarily equal ...
user avatar
7 votes
6 answers
17k views

How to find and delete duplicate files within the same directory?

I want to find duplicate files, within a directory, and then delete all but one, to reclaim space. How do I achieve this using a shell script? For example: pwd folder Files in it are: log.bkp log ...
user avatar
15 votes
11 answers
30k views

Remove all duplicate word from string using shell script

I have a string like "aaa,aaa,aaa,bbb,bbb,ccc,bbb,ccc" I want to remove duplicate word from string then output will be like "aaa,bbb,ccc" I tried This code Source $ echo "zebra ant spider ...
user avatar
  • 323
0 votes
1 answer
78 views

how to cut & paste and duplicate images

With the help of this forum I have managed to name my images with Exiftool by their unique date and time. My problem is that I have dozens of folders with hundreds of images in each folder combined ...
user avatar
2 votes
3 answers
969 views

Filter duplicate filenames with tar

I'm using tar like this, tar -cjpvf a.tar "$pattern1" "$pattern2" The problem is that pattern1 and pattern2 sometimes match the same file, but tar doesn't seem to filter these duplicates. e.g The ...
user avatar
  • 50.7k
1 vote
3 answers
5k views

How to remove duplicate source list on Debian 8.7

Am running Debian 8.7, after installing #letsencrypt following the tutorial on Digital Ocean. I couldn't update my operating system any more, If I run sudo apt-get update it displays the following ...
user avatar
0 votes
1 answer
56 views

How to delete lines in text file that contain three dupliate fields (strings sep by spaces)

I have a long text file as follows (generated from download links and downloaded filenames): file01 48976 5873987 48976 48976 thdshiesde file02 29876 589845637 29876 29876 uyjedrty 54 file03 38745 ...
user avatar
3 votes
1 answer
1k views

Remove duplicate lines from multiple files in a folder: Check only files within a range

I had a question about removing duplicate lines in multiple files and was provided with a useful script here: Remove duplicate lines from multiple JSON files while preserving file structure. The ...
user avatar
0 votes
1 answer
171 views

Dealing with duplicate hash data in text file - keeping one unique line and deleting duplicates

Thanks to this forum I was able to process a large text file containing filenames and hashes and end up with a list of files with identical hashes as shown below: file_35.txt ...
user avatar
0 votes
2 answers
2k views

Remove duplicate lines from multiple json files while preserving file structure

I have a folder with thousands of json files. The contents of each folder look something like this: file.1424-417982.json file.1424-417995.json file.1424-418013.json file.1424-418015.json (etc.) ...
user avatar
4 votes
1 answer
677 views

Text file containing filenames and hashes - extracting lines with duplicate hashes

I have generated a large text file containing filenames and sha-256 hashes using the format below - new line at the end of each line after the hashes. file_1.txt ...
user avatar
2 votes
2 answers
483 views

Deduplicate image files

[user@notebook foobar]$ ls *.jpg|wc -l 1959 [user@notebook foobar]$ cksum * | cut -d' ' -f-2 | sort | uniq -di | wc -l 698 [user@notebook foobar]$ There are many jpg files in a directory. Many of ...
user avatar
  • 943
0 votes
1 answer
264 views

Comparing files in awk/bash

I came across a problem and have no idea how to find the OPTIMAL solution. I have a list of files that looks like this: file1\0file2\0...fileX\0\0file(x+1)\0 Every name of file is separated by a \0 ...
user avatar
  • 191
0 votes
2 answers
159 views

How to duplicate an paragraph with one changed string

I'm looking for something that could copy a paragraph, change the user and insert it in same file. file before: user1 this is only a test of a lovely idea user2 this user shhould be ...
user avatar
4 votes
4 answers
5k views

Finding duplicate users on CentOS

I'm using the following code to find the duplicate username. However, it gives an error. #!/bin/bash cat /etc/passwd | cut -f1 -d":" | /bin/sort -n | /usr/bin/uniq -c |\ while read x; do [ -z "${...
user avatar
1 vote
1 answer
919 views

Find case duplicate files and rm uppercase

I have a number of image files that have duplicates across case (ie file.jpg File.jpg). I need a script like the following except I want to completely remove all uppercase. I know that fslint can do ...
user avatar
  • 13
5 votes
3 answers
4k views

Find and list duplicate directories

I have directory that has a number of sub-directories and would like to find any duplicates. The folder structure looks something like this: └── Top_Dir └── Level_1_Dir ├── standard_cat ...
user avatar
  • 51
7 votes
6 answers
6k views

How to delete all duplicate hardlinks to a file?

I've got a directory tree created by rsnapshot, which contains multiple snapshots of the same directory structure with all identical files replaced by hardlinks. I would like to delete all those ...
user avatar
  • 7,296
3 votes
0 answers
1k views

how to find duplicate folders inside main folder [closed]

I've moved a lot of files and folders from my old nas to my new smb share. The problem is that someone made entire copies of nearly every folder on that thing. And I need the space. I'm currently ...
user avatar
  • 141
5 votes
0 answers
2k views

Space efficiency of Btrfs reflinks vs hardlinks

A webpage referenced by the Wikipedia article on Btrfs claims that (emphasis mine) [Btrfs] reflinks have the same use as hardlinks, but are more space efficient. I thought that the opposite was ...
user avatar
0 votes
1 answer
2k views

Identifying duplicate fields and REMOVE both with awk

Previously, I asked the question: Identifying duplicate fields and print both with awk. I have a file with multiple columns and wanted to identify lines where specific column values (cols 3-6) had ...
user avatar
  • 349
1 vote
2 answers
1k views

Prints duplicate lines based on first column

I want to get lines where first column duplicates (I used number 32 becouse first column has always 32 characters). So far I've figured out: sort | uniq -d -w 32 filename The problem is that I want ...
user avatar
  • 51
1 vote
5 answers
143 views

Remove repetitive entries from a file

There is a file with a couple of identical and repetitive entries, which looks something like this: 123 abc nhjk 123 abc cftr 123 abc xdrt 123 def nhjk 123 def cftr 123 def xdrt If the combination ...
user avatar
  • 131
0 votes
1 answer
168 views

Keeping first instance of duplicates when column values are interchangeable

I have previously asked a question about keeping the first instance of duplicates and the solution provided has been working perfectly until now (see Keeping first instance of duplicates). However, I ...
user avatar
  • 349