Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a tab separated file that looks like this:

123   some text
123   some different text
334   some other text
341   more text

and I want to do two things. One is to order everything numerically (this is easy to do) and the other is to remove a line if it's number is already present. I.e. the output would look like this:

123   some text
334   some other text
341   more text

I tried getting a file of just the unique numbers, i.e.

123
334
341

and joining it with the original file with:

join -j 1 justNumbers.txt original.txt

but this gave me the original file back. Any ideas?

share|improve this question

2 Answers 2

up vote 6 down vote accepted

If you want to sort/test for uniqueness the first field specifically, and your system has the GNU coreutils version of sort, then I think you could just use

sort -nu file

viz.

$ sort -nu file
123   some text
334   some other text
341   more text


From info coreutils 'sort invocation'

The commands sort -u and sort | uniq are equivalent, but this equivalence does not extend to arbitrary sort options. For example, sort -n -u inspects only the value of the initial numeric string when checking for uniqueness, whereas sort -n | uniq inspects the entire line.

share|improve this answer
    
That should work with any POSIX sort, not only the GNU one. –  Stéphane Chazelas Jul 4 at 17:14
    
Note however that it you have one line starting with 0123 and another one starting with 123, it will keep only one of them (which may be what you want). Use sort -ubk1,1 | sort -n if it's not what you want. –  Stéphane Chazelas Jul 4 at 17:17
    
@StéphaneChazelas thanks (as usual) for your most knowledgeable input. –  steeldriver Jul 4 at 21:20

Try this awk:

awk '!($1 in a){a[$1];print}' file
123   some text
334   some other text
341   more text

This assumes that file was sorted.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.