2

Is it possible to remove only the AWD015F11 duplicate using Awk? I have a large file with one column. This happens every 24 rows.

I have tried counting the rows in search for a pattern that I could use. As you can see there are other duplicate rows but the problem is only with AWD015F11.

5000
5000
false
false
AWD015F11
425
25
34
1
5000
5000
320
320
OH,AWD015F
false
true
8
50
32
0
5
9
9
true
-34
0
false
false
AWD015F11
AWD015G21
3
  • a) Is every 24th row (after row 5) always going to contains this string, and hence will always be removed? (b) Do you have to use awk?
    – Sparhawk
    Jun 19, 2015 at 7:08
  • r.a) It is every 24 rows once AWD####### shows up. The output would be the basically showing only the first AWD015F11. r.b) It can also be sed or unix shell. Jun 19, 2015 at 7:16
  • 1
    I don't understand. Is the string to remove always going to be AWD015F11? Is it whatever is on the 24th column? How can the script know not to remove the duplicates you actually want? What is AWD#######? Is that supposed to represent AWD followed by any 7 characters? Does OH,AWD015F count? I don't see any cases of AWD and 7 characters. Please edit your question and clarify what you need. Show us a sample that reproduces the issue and show us the output you would like to see.
    – terdon
    Jun 19, 2015 at 12:53

3 Answers 3

2

If you can use PERL, use the code below to remove all duplicate from the file. It will print first unique AWD###### line, and the remaining following this anywhere in the file will not be printed.

#!/usr/bin/perl
%data=();
foreach $r ( <STDIN> ) {
    chomp($r);
    if($r =~ /^AWD[A-Za-z0-9]{6}$/){
        if(!exists($data{$r})){
            $data{$r} = 1;
            print "$r\n";
        }
    } else {
        print "$r\n";
    }
}

Link to ideone: http://ideone.com/0SFQQ4

I think this is what you want.

6
  • Thanks for this attempt to answer. I apologize for not explaining the part where 015F11 varies. Therefore, the only fix pattern is that every 24 rows, I need to remove a duplicate that starts with AWD######. Jun 19, 2015 at 8:39
  • @CurryMurtag is AWD###### of constant lenght. And I don't see how AWD015F11 AWD015G21 comes every 24 rows, Jun 19, 2015 at 8:43
  • Yes, the length is a constant. This is just a fraction of the file to illustrate the issue. Just imagine, once AWD015G21 shows up, 24 rows later, it shows up once again. Jun 19, 2015 at 8:54
  • @CurryMurtag I have changed my answer. Have a look at it. Jun 19, 2015 at 9:58
  • Thanks but I cannot get the script to run. Jun 23, 2015 at 6:33
0

With awk

awk '!/^AWD015F11$/ {print $0}; /^AWD015F11$/ && found<1 {found++; print $0}' foo

Example

$ cat foo | wc -l
30

$ awk '!/^AWD015F11$/ {print $0}; /^AWD015F11$/ && found<1 {found++; print $0}' foo | wc -l
29

$ cat foo
5000
5000
false
false
AWD015F11
425
25
34
1
5000
5000
320
320
OH,AWD015F
false
true
8
50
32
0
5
9
9
true
-34
0
false
false
AWD015F11
AWD015G21

$ awk '!/^AWD015F11$/ {print $0}; /^AWD015F11$/ && found<1 {found++; print $0}' foo
5000
5000
false
false
AWD015F11
425
25
34
1
5000
5000
320
320
OH,AWD015F
false
true
8
50
32
0
5
9
9
true
-34
0
false
false
AWD015G21
1
  • I like this approach. It does the job, but partially. Is it possible to get what awk is suppose to search, from another file? I mean, what's between /^#########$/ Jun 21, 2015 at 3:22
-1
n=$(set '' p n p n;printf "\n%b$@$@$@$@$@$@\n \c")
sed -ne"/^AWD015F11/{:n$n$n};s///;t" -ep <in

works with sed. It just does n;p 23 times after seeing ^AWD and then refuses to print same.

6
  • When trying this solution, I get: sed: -e expression #2, char 116: extra characters after command Jun 21, 2015 at 3:27
  • @Curry_Montag - i changed it to make it easier to enter on the command line. I suspect the issue either had to do with either the trailing newline that needed appending to $n's definition, or the shell doing psycho history expansion on a quoted ! - i really hate it when they do that. So i altered the quoting a little and expand $n's definition w/ a little set/printf macro in a subshell.
    – mikeserv
    Jun 21, 2015 at 4:13
  • This is a good option but I am afraid it needs some adjustments since it is removing all instances of whatever is between /^#########/ and it is only suppose to remove the second one Jun 23, 2015 at 6:47
  • @CurryMurtag - i didn't get the message you left the other day for whatever reason. I think i;ve got it to print only the first of any two paired occurrences. What is /^###.../?
    – mikeserv
    Jun 24, 2015 at 22:25
  • @CurryMurtag - just checked it out - it works just fine.
    – mikeserv
    Jun 24, 2015 at 22:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .