Tag Info

New answers tagged


for i in 799 800 do head -n"$i" >&"$((i&2|1))" done <infile 3>/dev/null The above code will send the first 799 lines of an lseek()able <infile to /dev/null, and the next 800 lines of same to stdout. If you want to prune those 800 lines for sequential uniques, just append |uniq. In that case, you might also do: sed ...


Costas' elegant answer in comment sed '800,1600 ! d' file


normal: echo "a b a b c c c" | tr ' ' '\n' a b a b c c c uniq : no two subsequent repeating lines echo "a b a b c c c" | tr ' ' '\n' | uniq a b a b c sorted echo "a b a b c c c" | tr ' ' '\n' | sort a a b b c c c sort -u : no two repeating lines echo "a b a b c c c" | tr ' ' '\n' | sort -u a b c sort / uniq: all distinct echo "a b a b c ...


Short version: uniq, without -u, makes every line of the output unique. uniq -u only prints every unique line from the input. Slightly longer version: uniq is for dealing with files that have lines duplicated, and only when those lines appear successively in the input. So, for its purposes, a unique line is one that is not duplicated immediately. (uniq ...


uniq POSIX spec described it clearly: -u Suppress the writing of lines that are repeated in the input. -u option make uniq not to print repeated lines. Most uniq implementations used bytes comparison, while GNU uniq used collation order to filter duplicated lines. So it can produce wrong result in some locales, example in en_US.UTF-8 locale: $ ...


uniq with -u skips any lines that have duplicates. Thus: $ printf "%s\n" 1 1 2 3 | uniq 1 2 3 $ printf "%s\n" 1 1 2 3 | uniq -u 2 3 Usually, uniq prints lines at most once (assuming sorted input). This option actually prints lines which are truly unique (having not appeared again).


I'd use perl and a hash. Something like: #!/usr/bin/perl use strict; use warnings; my %seen; while ( <> ) { print unless $seen{$_}++; } I think this'd one-liner-ify as: perl -ne 'print unless $seen{$_}++' data.txt (Or cat data into it). This works on getting unique whole lines - you can also use split or regular expressions ...


$ cat /var/log/file | sort | uniq


That's the job for uniq: LC_ALL=C uniq file GNU uniq in some locales can report first of sequences of lines that sort the same. Using LC_ALL=C forced bytes comparison behavior, give you persistent result.


I prefer to use sort | uniq because when I try to use the -u (eliminate duplicates) option to remove duplicates involving mixed case strings, it is not that easy to understand the result. Note: before you can run the examples below, you need to simulate the standard C collating sequence by doing the following: LC_ALL=C export LC_ALL For example, if I ...

Top 50 recent answers are included