3

I have an input file like below.

file.txt
14 2.692022 author1 0
14 2.57826 author2 1
229 2 author3 2
323 2 author4 3
514 2.0625 author5 4

I am trying to split the file with the duplicated values as one file and the rest of values as another file. So, for the above input, I am trying to generate files like,

file1.txt
14 2.692022 author1 0
14 2.57826 author2 1

file2.txt
229 2 author3 2
323 2 author4 3
514 2.0625 author5 4

I am not able to figure out how to use awk for this purpose.

awk '{print $1}' file.txt | uniq -d > file1.txt

The above command gives me only the duplicated values in a file. Also, it gives me only the field1 but not all the 3 fields.

3

awk supports direct file redirection in its syntax.

awk 'BEGIN{prev=""} {if($1==prev) { print $0 > "duplicates.txt"} else { print $0 > "originals.txt"}; prev=$1; }' file.txt

I assumed the file is already sorted. If it's not, it gets much harder.

EDIT: is this better?

awk 'BEGIN{prev=""; prevfull=""}
$1==prev{ if(prevfull!=""){print prevfull > "duplicates.txt"; prevfull=""}; print > "duplicates.txt"}
$1!=prev{ if(prevfull!=""){print prevfull > "originals.txt"}; prevfull=$0}
{prev=$1; } END{print prevfull}' file.txt

It can definitely be done better (the ifs are definitely not optimal) but it shows the general idea.

  • Thanks. The file is already sorted. I will check this command. – Ramesh Mar 11 '14 at 22:29
  • I see some values are copied to originals.txt though they are duplicate values. They are basically present in both the files. – Ramesh Mar 11 '14 at 22:33
  • Oh I think I misunderstood. I put the first instance in originals and the subsequent into duplicates. It can be fixed :) – orion Mar 11 '14 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.