1

I have 2 files, each containing the same amount of 4-lines records, in the same order:

fileA:

record1 line1=header
record1 line2 X <= this is different but should be ignored
record1 line3 id ABC
record1 line4
record2 line1=header
record2 line2
record2 line3 id DEF <= this is different
record2 line4

fileB:

record1 line1=header
record1 line2 Y <= this is different but should be ignored
record1 line3 id ABC
record1 line4
record2 line1=header
record2 line2
record2 line3 id GHI <= this is different
record2 line4

For each record, I want to compare its line3 between 2 files and if line3 are different, save the whole record (lines1-4) of fileB; in the example above, record1 will be ignored and record2 saved. I have basic knowledge of diff and not sure if it is doable at all. First, I don't know how to compare only every 3rd line and ignore the others; second, -C defines symmetrical context, i.e. equal number of lines before and after the difference...

UPD. Initially I had a mistake in my examples: blank line between records which I don't have in my real files. I apologize for this. Based on @stteldriver's answer, I have the following solution:

awk '
NR%4==3 {
lineA3=$0; 
getline lineB1 < "fileB"; 
getline lineB2 < "fileB"; 
getline lineB3 < "fileB"; 
getline lineB4 < "fileB";  
if (lineA3 != lineB3) {printf "%s\n%s\n%s\n%s\n", lineB1,lineB2,lineB3,lineB4;}
}' fileA

It works perfectly! Though the code is quite ugly (I'm only starting to learn awk!), will be grateful if you can optimize it.

  • 1
    Your updated code above is sound, although uniquely tailored to your two input file format. You could perhaps simplify it a bit, but it is more a matter of style. The last test could be written: lineA3 != lineB3 && {printf "%s\n%", lineB1,lineB2,lineB3,lineB4} As you can see the if (...) construct is implicit and the printf method uses the same format and applies it to any input fed to it until exhausted. --- Steeldriver's array-based code is fine of course, although more complex. – Cbhihe Mar 17 at 9:15
3

For your updated input, based strictly on 4-line records, you can use modulo arithmetic to maintain arrays of the current records, and check the 3rd lines for a match every 4th line:

$ awk '
    {a[FNR%4] = $0; getline b[FNR%4] < "fileB"} 
    !(FNR%4) && b[3] != a[3] {
      for(i=0;i<4;i++) print b[i%4]
    }
  ' fileA
record2 line1=header
record2 line2
record2 line3 id GHI <= this is different
record2 line4

(note that one really should check the return value of the getline command, and do something sensible if it fails).


For your originally-posted input, you could have used paragraph mode:

$ awk -vRS= -F'\n' '{A3 = $3}; getline "fileB" > -1 && $3 != A3' fileA
record2 line1=header
record2 line2
record2 line3 id DEF <= this is different
record2 line4

The empty RS causes whole blank-line separated records to be read, for both normal processing (input from fileA) and for getline (input from fileB). Setting the field separator to newline (\n) then allows us to save the whole line $3 from one and compare to the other. If they are not equal, the default print outputs $0 (which is the whole record from the getline of fileB).

  • thanks a lot @steeldriver! I'm so sorry, I made a mistake in my example: in fact my records are not separated by blank lines, it goes continuously 4-4-4-4... Do you have any idea for the solution that would be based only on the line number? (but in any case please don't delete the current solution: I learned a lot from it). I apologize again for the confusion. – Vasily A Mar 16 at 1:50
  • @VasilyA I have added a different approach for your updated case – steeldriver Mar 16 at 2:48
  • awesome, thanks! (I can't say I understand completely how it works :) but it works) – Vasily A Mar 16 at 13:15
  • why print b[i%4] instead of just print b[i] ? – Cbhihe Mar 17 at 9:17
  • 1
    @Cbhihe because the arrays are constructed using FNR%4, line4 becomes b[0] - using b[i%4] in the print statement ensures the lines of the record are output in the correct order (1,2,3,0). – steeldriver Mar 17 at 12:27
3

Here's a Python script that should do what you want:

#!/usr/bin/env python3
# -*- encoding: utf-8 -*-
"""compare.py"""

import sys

file_A = open(sys.argv[1], "r")
file_B = open(sys.argv[2], "r")

records_A = file_A.read().split("\n\n")
records_B = file_B.read().split("\n\n")

for record_A, record_B in zip(records_A, records_B):
    lines_A = record_A.split("\n")
    lines_B = record_B.split("\n")
    if lines_A[2] != lines_B[2]:
        print("{}\n".format(record_B))

You would run it like this:

python compare.py fileA fileB

When I run it on your example data, I get the desired output:

record2 line1=header
record2 line2
record2 line3 id GHI <= this is different
record2 line4
  • thank you @igal, I really appreciate the input! Ideally, I wanted some sort of shell-only based solution, awk is closer to that. But maybe your solution will work faster for the big files I have... I will make a benchmark and add a comment about it. – Vasily A Mar 16 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.