Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have an input file (input.txt) like below.

id1      id2       name    weight 
53723848 12651711 timburnes 1.36667
53530214 12651711 timburnes 1.51191
53723848 53530214 timburnes 1.94
764157 52986038 ericcartman 0.861145
56797854 764157 ericcartman 1.35258
56797854 52986038 ericcartman 1.73781

Note that the first line is not part of the actual file, I have added it here for clarity.

I am trying to extract the values of the id1 and id2 to 2 separate files named unique.txt and duplicate.txt.

If my weight column value is greater than 1.5, it means I have duplicate ids. In this case, I will move the id1 value to unique.txt file and id2 value to duplicate.txt file.

If my weight column is less than 1.5, it means I do not have duplicate values. So, in this case, I will move both id1 and id2 to unique.txt file.

So for the above input, I am expecting the output as,

For unique.txt file,

53723848 timburnes
764157 ericcartman
56797854 ericcartman

For duplicate.txt file,

12651711 timburnes
53530214 timburnes
52986038 ericcartman

I can find out the duplicates using the below code.

To get the values greater than 1.5 based on 4th column,

awk -F" " '$4 >= 1.5 { print $1" " $2" " $3" " $4}' file1.txt > Output.txt

Now, for values greater than 1.5, I can use the below code to merge the duplicate ids based on their names.

  perl -ane 'foreach(@F[0..1]){$k{$F[2]}{$_}++}
           END{
                foreach $v (sort keys(%k)){
                    print "$_ " foreach(keys(%{$k{$v}})); 
                    print "$v\n"
                }; 
            } ' Output.txt

However, I am not able to get the output in the way I like in the above approach.

EDIT:

I am running the command for my input as below.

awk '{
      if ($4 > 1.5) { 
          if (++dup[$2] == 1)  print $2, $3 > "duplicate.txt"
      } 
      else
          if (++uniq[$1] == 1) print $1, $3 > "unique.txt" 
}' << END
17412193 43979400 ericcartman 2.16667
21757330 54678379 andrewruss 0.55264
END 

I am getting the output as,

-bash-3.2$ cat unique.txt
21757330 a.andreev
-bash-3.2$ cat duplicate.txt
43979400 ericcartman

However, the output I am expecting is,

cat unique.txt
17412193 ericcartman
21757330 andrewruss
54678379 andrewruss
cat duplicate.txt
43979400 ericcartman
share|improve this question
    
What happen if $4 less than 1.5? –  cuonglm May 6 at 16:22
    
@Gnouc, if $4 is less than 1.5, I will add them to unique.txt file. Also, before inserting something to either unique.txt or duplicate.txt file, I will have to check if the value is already present in the file. If it is already present, I need not insert it again. –  Ramesh May 6 at 16:23
    
What "add them" means? What happen with $1 and $2 when $4 is less than 1.5? –  cuonglm May 6 at 16:44
    
@Gnouc, add them means I am appending the values to the unique.txt file. So, suppose if $4 is less than 1.5, I will append $1 and $2 to unique.txt file. –  Ramesh May 6 at 16:48
    
"I will have to check if the value is already present" -- which value? Just the ID or the ID/name pair? –  glenn jackman May 6 at 16:48

3 Answers 3

up vote 7 down vote accepted

Here is awk solution:

$ awk '
    $4 < 1.5 {
      uniq[$1] = $3;
      uniq[$2] = $3;
      next;
  }
  {
      uniq[$1] = $3;
      dup[$2] = $3;
      delete uniq[$2];
  }
  END {
    print "--unique.txt--";
    for(i in uniq) {
        print i,uniq[i]
    }
    print "";
    print "--duplicate.txt--";
    for(i in dup) {
        print i,dup[i]
    }
    }' file
--unique.txt--
764157 ericcartman
56797854 ericcartman
53723848 timburnes

--duplicate.txt--
53530214 timburnes
52986038 ericcartman
12651711 timburnes

With your second example:

$ awk '
    $4 < 1.5 {
      uniq[$1] = $3;
      uniq[$2] = $3;
      next;
  }
  {
      uniq[$1] = $3;
      dup[$2] = $3;
      delete uniq[$2];
  }
  END {
    print "--unique.txt--";
    for(i in uniq) {
        print i,uniq[i]
    }
    print "";
    print "--duplicate.txt--";
    for(i in dup) {
        print i,dup[i]
    }
    }' << END
> 17412193 43979400 ericcartman 2.16667
> 21757330 54678379 andrewruss 0.55264
END
--unique.txt--
21757330 andrewruss
54678379 andrewruss
17412193 ericcartman

--duplicate.txt--
43979400 ericcartman
share|improve this answer
    
Thanks a lot. Is it possible for me to get the values to file? If I use it now, I am getting the output printed in the console and so am not able to check it. :) –  Ramesh May 6 at 17:30
    
Replace print i,uniq[i] by print i,uniq[i] >> "unique.txt". Do the same thing with dup. A note that the output is not sorted. –  cuonglm May 6 at 17:36
    
I tried that only. I get some syntax error. –  Ramesh May 6 at 17:38
    
What line causes syntax error? –  cuonglm May 6 at 17:41
    
Sorry, I was using single quotes for file. I think it should work now with double quotes. Let me check it :) –  Ramesh May 6 at 17:44
$ awk '{
      if ($4 > 1.5) { 
          if (++dup[$2] == 1)  print $2, $3 > "duplicate.txt"
      } 
      else
          if (++uniq[$1] == 1) print $1, $3 > "unique.txt" 
}' << END
53723848 12651711 timburnes 1.36667
53530214 12651711 timburnes 1.51191
53723848 53530214 timburnes 1.94
764157 52986038 ericcartman 0.861145
56797854 764157 ericcartman 1.35258
56797854 52986038 ericcartman 1.73781
END

$ cat unique.txt 
53723848 timburnes
764157 ericcartman
56797854 ericcartman

$ cat duplicate.txt 
12651711 timburnes
53530214 timburnes
52986038 ericcartman
share|improve this answer
    
Please see the edit of my question. I ran your command and pasted the output that am getting. –  Ramesh May 6 at 17:09

Here's a Perl one:

perl -lane '$F[3]>1.5 ? print STDERR "$F[1] $F[2]" : print STDOUT "$F[0] $F[2]"'\
 input.txt 2> duplicate.txt > unique.txt

I am not checking for duplicates here, if I understand your question correctly, you have already done so and whether something is a dupe or not depends on the value of the last field. If I am misunderstanding something please let me know and I will update this.

The code above produces

$ cat duplicate.txt 
12651711 timburnes
53530214 timburnes
52986038 ericcartman

$ cat unique.txt 
53723848 timburnes
764157 ericcartman
56797854 ericcartman
share|improve this answer
    
You solution is only right for these input, it provides the right desired output but with wrong OP's logic. The OP wants 1) if $4 < 1.5, then write both entry ($1 $3) and ($2 $3) to unique.txt --- 2) if $4 >= 1.5, then write ($1 $3) to unique.txt and write ($2 $3) to duplicate.txt, if $2 was in unique.txt, remove its entry in unique.txt. –  cuonglm May 6 at 17:32
    
@Gnouc you may well be right. I honestly have no idea what the OP is asking for at this point. The question is very confusing and I see no relation between the input and the desired output(s). I'll wait for Ramesh to clarify. –  terdon May 6 at 17:38
    
@terdon, am really sorry for pretty confusing question :) I will try and update it clearly in some time. Again, I am sorry for not being clearer :) –  Ramesh May 6 at 18:27
    
@Ramesh relax! As I said in my previous comment, I have a flu at the moment so it is quite possible that I'm just being obtuse. Don't worry about it :) –  terdon May 6 at 18:28
    
@terdon, oh no. Hope you get well soon :) Is it MERS virus? –  Ramesh May 6 at 18:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.