Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
#!/bin/awk
BEGIN {
        while(getline var < compareTo > 0)
        {
                orderIds[var]=var;
        }
}
{
        if(orderIds[$0] == "")
        {
                print $0;
        }
}

Running as

awk -v compareTo="ids.log.remote" -f sample.awk ids.log.local

This is working, but instead of using associative arrays ( like HashMap ), is there anything like a HashSet in awk?

I got the timings

bash-3.2$ time grep -xFvf ids.log.local ids.log.remote > /dev/null

real    0m0.130s
user    0m0.127s
sys     0m0.002s
bash-3.2$ time grep -xFvf ids.log.local ids.log.remote > /dev/null

real    0m0.126s
user    0m0.125s
sys     0m0.000s
bash-3.2$ time grep -xFvf ids.log.local ids.log.remote > /dev/null

real    0m0.131s
user    0m0.128s
sys     0m0.002s
bash-3.2$ time awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.local ids.log.remote > /dev/null

real    0m0.053s
user    0m0.051s
sys     0m0.003s
bash-3.2$ time awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.local ids.log.remote > /dev/null

real    0m0.052s
user    0m0.051s
sys     0m0.001s
bash-3.2$ time awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.local ids.log.remote > /dev/null

real    0m0.053s
user    0m0.051s
sys     0m0.002s
bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log.remote > /dev/null

real    0m0.066s
user    0m0.060s
sys     0m0.006s
bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log.remote > /dev/null

real    0m0.065s
user    0m0.058s
sys     0m0.008s
bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log.remote > /dev/null

real    0m0.061s
user    0m0.053s
sys     0m0.007s

@Dimitre Radoulov Looks like your awk is faster. Thanks.

share|improve this question
    
What exactly are you trying to do? –  Kevin Feb 2 '12 at 20:50
    
In your example, orderIds[var]=var is storing var twice; once as the the index, and then again as the value held by that array element. This is not necessary and inhibits the detection of blank lines in the main section, ie.when var == "". In the BEGIN section, you can set orderIds[var]=1.. The 1 is just a flag to indicate that this particular index (var) has been encountered in "ids.log.remote" –  Peter.O Feb 2 '12 at 21:09
    
@Peter.O yes, I thought about storing 1 and I thought both are integer values and it requires same amount of storage to store ints. So, I left it as var, instead of a flag like 1. –  user14039 Feb 2 '12 at 21:12
    
@tech_learner, it's not true that both are integer values, rather var here has a string value. But that string isn't going to be freed anyway, since you're using it as the array index, so you're only using up the space of an active pointer. Probably the same length as an integer value, but it's better hygiene not to think of storing a pointer as "storing an int". I also find it more natural to use orderIds[var]=1 here; that's a familiar pattern for implementing sets in terms of assoc arrays that others will more immediately recognize when reading your code. –  dubiousjim Apr 19 '12 at 19:34
add comment

1 Answer

up vote 3 down vote accepted

I believe this is the most efficient way to do this in awk:

awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.remote ids.log.local

You may try with grep too:

grep -xFVf ids.log.remote ids.log.local 
share|improve this answer
    
I changed the script similar to yours. Now it looks like modified script is faster. No idea why #!/bin/awk BEGIN { while(getline var < compareTo > 0) { orderIds[var]; } } { if(!($0 in orderIds)) { print $0; } } bash-3.2$ time awk 'NR == FNR { orderIds[$0]; next } !($0 in orderIds) ' ids.log.local ids.log > /dev/null real 0m0.052s user 0m0.050s sys 0m0.001s bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log > /dev/null real 0m0.036s user 0m0.035s sys 0m0.001s –  user14039 Feb 2 '12 at 21:35
1  
I suppose it's mostly because every time you're referencing a non-existing key (i.e. orderIds[$0] when $0 is not currently in orderIds) you're actually creating a new entry in the array (with a null string value). Try to test the number of elements in the orderIds array after all input has been read (in an END block) with your code and compare it with my script - with your script you'll end up with more entries (one for every printed line). Out of curiosity, you may try the script with mawk (if you have it), it could be even faster. –  Dimitre Radoulov Feb 2 '12 at 21:37
1  
Another contributing factor to the time difference between the two awk versions could be that accessing the data component of the array element simply takes longer than just checking for the existence of the key.. –  Peter.O Feb 2 '12 at 23:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.