Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Is it possible to remove only the AWD015F11 duplicate using Awk? I have a large file with one column. This happens every 24 rows.

I have tried counting the rows in search for a pattern that I could use. As you can see there are other duplicate rows but the problem is only with AWD015F11.

5000
5000
false
false
AWD015F11
425
25
34
1
5000
5000
320
320
OH,AWD015F
false
true
8
50
32
0
5
9
9
true
-34
0
false
false
AWD015F11
AWD015G21
share|improve this question
    
a) Is every 24th row (after row 5) always going to contains this string, and hence will always be removed? (b) Do you have to use awk? –  Sparhawk Jun 19 at 7:08
    
r.a) It is every 24 rows once AWD####### shows up. The output would be the basically showing only the first AWD015F11. r.b) It can also be sed or unix shell. –  Curry Murtag Jun 19 at 7:16
1  
I don't understand. Is the string to remove always going to be AWD015F11? Is it whatever is on the 24th column? How can the script know not to remove the duplicates you actually want? What is AWD#######? Is that supposed to represent AWD followed by any 7 characters? Does OH,AWD015F count? I don't see any cases of AWD and 7 characters. Please edit your question and clarify what you need. Show us a sample that reproduces the issue and show us the output you would like to see. –  terdon Jun 19 at 12:53

3 Answers 3

If you can use PERL, use the code below to remove all duplicate from the file. It will print first unique AWD###### line, and the remaining following this anywhere in the file will not be printed.

#!/usr/bin/perl
%data=();
foreach $r ( <STDIN> ) {
    chomp($r);
    if($r =~ /^AWD[A-Za-z0-9]{6}$/){
        if(!exists($data{$r})){
            $data{$r} = 1;
            print "$r\n";
        }
    } else {
        print "$r\n";
    }
}

Link to ideone: http://ideone.com/0SFQQ4

I think this is what you want.

share|improve this answer
    
Thanks for this attempt to answer. I apologize for not explaining the part where 015F11 varies. Therefore, the only fix pattern is that every 24 rows, I need to remove a duplicate that starts with AWD######. –  Curry Murtag Jun 19 at 8:39
    
@CurryMurtag is AWD###### of constant lenght. And I don't see how AWD015F11 AWD015G21 comes every 24 rows, –  Rakholiya Jenish Jun 19 at 8:43
    
Yes, the length is a constant. This is just a fraction of the file to illustrate the issue. Just imagine, once AWD015G21 shows up, 24 rows later, it shows up once again. –  Curry Murtag Jun 19 at 8:54
    
@CurryMurtag I have changed my answer. Have a look at it. –  Rakholiya Jenish Jun 19 at 9:58
    
Thanks but I cannot get the script to run. –  Curry Murtag Jun 23 at 6:33

With awk

awk '!/^AWD015F11$/ {print $0}; /^AWD015F11$/ && found<1 {found++; print $0}' foo

Example

$ cat foo | wc -l
30

$ awk '!/^AWD015F11$/ {print $0}; /^AWD015F11$/ && found<1 {found++; print $0}' foo | wc -l
29

$ cat foo
5000
5000
false
false
AWD015F11
425
25
34
1
5000
5000
320
320
OH,AWD015F
false
true
8
50
32
0
5
9
9
true
-34
0
false
false
AWD015F11
AWD015G21

$ awk '!/^AWD015F11$/ {print $0}; /^AWD015F11$/ && found<1 {found++; print $0}' foo
5000
5000
false
false
AWD015F11
425
25
34
1
5000
5000
320
320
OH,AWD015F
false
true
8
50
32
0
5
9
9
true
-34
0
false
false
AWD015G21
share|improve this answer
    
I like this approach. It does the job, but partially. Is it possible to get what awk is suppose to search, from another file? I mean, what's between /^#########$/ –  Curry Murtag Jun 21 at 3:22
n=$(set '' p n p n;printf "\n%b$@$@$@$@$@$@\n \c")
sed -ne"/^AWD015F11/{:n$n$n};s///;t" -ep <in

works with sed. It just does n;p 23 times after seeing ^AWD and then refuses to print same.

share|improve this answer
    
When trying this solution, I get: sed: -e expression #2, char 116: extra characters after command –  Curry Murtag Jun 21 at 3:27
    
@Curry_Montag - i changed it to make it easier to enter on the command line. I suspect the issue either had to do with either the trailing newline that needed appending to $n's definition, or the shell doing psycho history expansion on a quoted ! - i really hate it when they do that. So i altered the quoting a little and expand $n's definition w/ a little set/printf macro in a subshell. –  mikeserv Jun 21 at 4:13
    
This is a good option but I am afraid it needs some adjustments since it is removing all instances of whatever is between /^#########/ and it is only suppose to remove the second one –  Curry Murtag Jun 23 at 6:47
    
@CurryMurtag - i didn't get the message you left the other day for whatever reason. I think i;ve got it to print only the first of any two paired occurrences. What is /^###.../? –  mikeserv Jun 24 at 22:25
    
@CurryMurtag - just checked it out - it works just fine. –  mikeserv Jun 24 at 22:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.