4

I want to remove duplicate lines from /etc/fstab, so I did this:

 awk '!NF || !seen[$0]++'   /etc/fstab > /etc/fstab.update

UUID=3de0d101-fba7-4d89-b038-58fe07295d96 /grid/sdb ext4 defaults,noatime 0 0
UUID=683ed0b3-51fe-4dc4-975e-d56c0bbaf0bc /grid/sdc ext4 defaults,noatime 0 0
UUID=1cf79946-0ba6-4cd8-baca-80c0a2693de1 /grid/sdd ext4 defaults,noatime 0 0
UUID=fa9cc6e8-4df8-4330-9144-ede46b94c49e /grid/sde ext4 defaults,noatime 0 0
UUID=3de0d101-fba7-4d89-b038-58fe07295d96   /grid/sdb                      ext4 defaults,noatime 0 0
UUID=683ed0b3-51fe-4dc4-975e-d56c0bbaf0bc   /grid/sdc                      ext4 defaults,noatime 0 0

But as we can see, the last two lines are the same with the first two lines, but last two lines are with spaces.

Is it possible to ignore the space and remove the duplicate lines anyway?

0
13

Force the rebuild of the record with $1=$1! This squeezes all contiguous spaces into a single one.

awk '{$1=$1};!NF||!seen[$0]++'
0
6

Use tr to replace tab with space and squeeze repeats (-s):

 tr -s $'\t' ' ' < /etc/fstab | awk '!NF || !seen[$0]++' > /etc/fstab.update
5

Use this Perl one-liner to treat any amount of whitespace as a single blank:

perl -lane 'print unless $seen{"@F"}++' in.txt > out.txt

If you want to ignore whitespace completely, use:

perl -lane '$s = join "", @F; print unless $seen{$s}++' in.txt > out.txt

The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-n : Loop over the input one line at a time, assigning it to $_ by default.
-l : Strip the input line separator ("\n" on *NIX by default) before executing the code in-line, and append it when printing.
-a : Split $_ into array @F on whitespace or on the regex specified in -F option.

SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.