How can we delete duplicate lines only seen in lines X to Y and change the file in-place?
For instance, if I just want to delete duplicate lines from line 10 to 20.
8
-
1are the targeted lines already sorted, and if not, can they be as a result?– Jeff Schaller ♦Commented Aug 7, 2016 at 18:16
-
duplicates within 10-20 lines or trough all file?– CostasCommented Aug 7, 2016 at 19:21
-
I'm confused. Your comment on sjsam's answer indicates duplicates just within lines 10-20, but your latest comment says the duplicates can exist anywhere in the file.– Jeff Schaller ♦Commented Aug 7, 2016 at 20:43
-
@Costas sorry I meant only lines 10-20. But I have 5 files and I want all duplicates between line 10-20 be deleted.– Parsa SametCommented Aug 7, 2016 at 21:50
-
@JeffSchaller Sorry, I made mistake, edited my answer to Costas.– Parsa SametCommented Aug 7, 2016 at 21:51
|
Show 3 more comments
4 Answers
With GNU awk (4.1.0 or above for the inplace extension¹):
gawk -i /usr/share/awk/inplace.awk '
NR >= 10 && NR <= 20 {
if ($0 in seen) next
seen[$0]
}
{print}' ./file
Or with perl:
perl -ni -e 'print if $. < 10 or $. > 20 or !$seen{$_}++' ./file
To process several files:
gawk -i /usr/share/awk/inplace.awk '
BEGINFILE{delete seen}
FNR >= 10 && FNR <= 20 {
if ($0 in seen) next
seen[$0]
}
{print}' ./*.txt
Or with perl:
perl -ni -e '
print if $. < 10 or $. > 20 or !$seen{$_}++;
if (eof) {close ARGV; undef %seen}' ./*.txt
¹ do not use -i inplace as gawk tries to load the inplace extension (as inplace or inplace.awk) from the current working directory first, where someone could have planted malware. The path of the inplace extension supplied with gawk may vary with the system, see the output of gawk 'BEGIN{print ENVIRON["AWKPATH"]}'
answered Aug 8, 2016 at 11:18
awk is your friend
awk '{
if(NR>=10 && NR<=20)
{
if($0 in record){
next
}else{
print;
record[$0];
}
}
else{
print
}
}' file > temp && mv temp file
-
1Indentation makes scripts more readable. Also the
{if(and)are not needed on the first line (theNR>=10 && NR<=20"pattern" is still required).awkscripts consist of a series of pattern {action}; statements, either of which are optional ("pattern" defaults to true ("condition" would probably be a better name for it than "pattern"), and "action" defaults to print`).– casCommented Aug 8, 2016 at 10:52
If OP need to remove lines which duplicates just within 10-20 lines:
sed -i '
:a; 10,19!b; N; s/\(^\|\n\)\([^\n]*\)\n\(\(.\+\n\|\)\2$\)/\1\3/; ba
' file1 file2 ...
-
1It gives the wrong result on the output of
printf '%s\n' {1..10} T TESTfor instance. It also assumes GNUsed(and thatPOSIXLY_CORRECTbe not in the environment).. Commented Aug 8, 2016 at 11:39 -
The very same tricks applied in the Perl-based answers can also be used to shorten the Awk code, and it ends up smaller and cleaner:
awk 'NR < 10 || NR > 20 || !seen[$0]++'
^ ^ ^ ^
| | | |
| \__________\___________\______ no sigil noise
|
\_ no options here to remember
(unless we want that Gawk inplace semantics)
The counters will not overflow because the range is restricted to ten lines—and GNU Awk has bignum integers anyway.
