5

I want to compare two files to check for differences between them, fileA and fileB. fileA is like a template file and fileB is the file which I want to compare to it. Anytime I find a difference I want to output that difference to fileC.

The difficulty is that fileA and fileB contain certain (not all) lines which have some data which will always be different - time, date and a randomnly generated id code. However, I do not want to output lines to fileC where the only difference is time, date and id code.

So what I would like to do is to remove the time, date and id code from any lines where they occur in fileB (I can do this manually in fileA) and then do a comparison against fileB, outputting the different lines to fileC.

Note that the text to remove always follows specific patterns. So I can find the text using grep with those patterns, but I don't know how to remove it...

Here is an example of the two files to show what I mean:

  • fileB

    qaqa rara
    abc 10:12:25 08/20/2014 123456 def
    ghi fff ddd
    jkl 09:20:40 08/20/2014 978645 dfdf gggg
    
  • fileA

    qaqa rara
    abc 10:32:15 07/15/2014 121456 xxx
    ghi eee ddd
    jkl 10:01:22 07/15/2014 971645 dfdf gggg
    

I want to find the difference between the above two files, disregarding the time (e.g 10:12:25), date (e.g. 08/20/2014) or id code (e.g. 123456) and output the differences to fileC

The two lines which are different, therefore are lines 2 and 3. Line 1 is the same for both files. Line 4 is the same for both files when the time, date and id info has been removed.

1
  • 1
    Please give us example input.
    – cuonglm
    Aug 20, 2014 at 10:15

4 Answers 4

3

If your timestamps are consistently formated, you could strip them off (with sed, for example) before processing the files with whatever differencing method, e.g.

diff <(sed -E 's|[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{2,4} [0-9]{1,} ||' fileA) <(sed -E 's|[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{2,4} [0-9]{1,} ||' fileB)

Testing on your supplied input files:

$ diff \
<(sed -E 's|[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{2,4} [0-9]{1,} ||' fileA) \
<(sed -E 's|[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{2,4} [0-9]{1,} ||' fileB)
2,3c2,3
< abc xxx
< ghi eee ddd
---
> abc def
> ghi fff ddd
6
  • I tried that and get the error message "Missing name for redirect".
    – didjek
    Aug 20, 2014 at 12:29
  • Apologies - my cut'n'paste had included additional shell prompt characters > at the start of the lines - please try with the corrected version above Aug 20, 2014 at 12:45
  • I tried it but I still see the same "Missing name for redirect" message.
    – didjek
    Aug 20, 2014 at 13:50
  • 1
    Red Hat 6.4 and csh; I changed to bash and your suggestion worked, thanks! I will check it on the original file and mark it as solved if it works on that.
    – didjek
    Aug 20, 2014 at 14:01
  • 1
    For better legibility, I'd write trim() { sed...; }; diff <(trim < fileA) <(trim < fileB). Aug 20, 2014 at 15:21
1

The easiest command to get result would be the below

$ diff <(tr -s "[0-9],:,/" " " < fileA) <(tr -s "[0-9],:,/" " " < fileB)

The command is very straight forward and there is no complex regular expression as well.

Sample output will be as below

2,3c2,3
< abc xxx
< ghi eee ddd
---
> abc def
> ghi fff ddd

Hope this is what you want.

1
diff \
<(sed -r 's\[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{4} [0-9]{6} \\' fileA) \
<(sed -r 's\[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{4} [0-9]{6} \\' fileB) \
| egrep '^> ' | sed -r 's/^> //' > fileC

Explanation

Strip the irrelevant portions as given in the OP's question from fileA and fileB and feed this into diff.

diff will output the changed sections with a preceeding "> ", so ignore everything else except the changes.

Finally strip the leading "> " from the output and store that in fileC as per the question.

I originally did it slightly differently but I just noticed that the files could vary on the irrelevant section so it has to be pre-stripped not post-stripped else diff would output info that hasn't actually changed when considering the relevant portions only.

Given the example input, cat fileC gives:

abc def
ghi fff ddd

The sed command is searching for the supplied regular expression which describes the irrelevant data, and replacing that with an empty string - ie, it deletes it.

4
  • I'm unsure how to use the first answer you provided. Here is what I tried to check for differences regardless of the time value: codediff newfile.txt otherfile.txt | egrep '^> [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}' | sed 's/^> //'code - however it did not produce any result, so I guess I'm doing something wrong. T
    – didjek
    Aug 20, 2014 at 11:53
  • Now that you gave an example, I've put in the exact code you need above. Aug 20, 2014 at 13:36
  • @IanMacintosh - you can break down what your sed is doing a bit, you don't have to do an entire recap of the man pages but braiam's right. We like the A's here to be self contained explanations, within reason 8-).
    – slm
    Aug 20, 2014 at 13:50
  • Good point @slm. I'll make the answer a bit more verbose. Aug 20, 2014 at 13:53
0
{   paste -d\| /dev/fd/3 /dev/fd/4 |
    sed '/\([^ ]*\) [0-9:/ ]*\(.*\)|\1 .*\2/d;=' |
    sed 'N;s/\(\n\)\(.*\)|/:\tFILEA: \2\1\tFILEB: /'
} 3<<\FILEA 4<<\FILEB
qaqa rara
abc 10:12:25 08/20/2014 123456 def
ghi fff ddd
jkl 09:20:40 08/20/2014 978645 dfdf gggg
FILEA
qaqa rara
abc 10:32:15 07/15/2014 121456 xxx
ghi eee ddd
jkl 10:01:22 07/15/2014 971645 dfdf gggg
FILEB

OUTPUT

2:      FILEA: abc 10:12:25 08/20/2014 123456 def
        FILEB: abc 10:32:15 07/15/2014 121456 xxx
3:      FILEA: ghi fff ddd
        FILEB: ghi eee ddd

You needn't be rid of the time and date - they're no great hurdle so long as the characters that compose them are reliable.

In the above pipeline paste first appends the corresponding line from FILEB to the tail of each of FILEA's lines with a single | separator then prints the results to stdout.

sed picks up the stream and compares:

  • the first sequence of 0 or more characters that are not space (referenced as \1)

  • all characters that occur between the following sequences : (referenced as \2)

    • at least a single <space> character then 0 or more of any of the following:

    • <space> characters

    • <digit> characters

    • <:colon> characters

    • </slash> characters

  • up to but not including the last occurring | character on the line

...with |\1.*\2. If they match sed deletes the line. If not it prints the line preceded by its line number.

The final sed process just pretties the output (I hope).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .