I want to compare two files to check for differences between them, fileA and fileB. fileA is like a template file and fileB is the file which I want to compare to it. Anytime I find a difference I want to output that difference to fileC.

The difficulty is that fileA and fileB contain certain (not all) lines which have some data which will always be different - time, date and a randomnly generated id code. However, I do not want to output lines to fileC where the only difference is time, date and id code.

So what I would like to do is to remove the time, date and id code from any lines where they occur in fileB (I can do this manually in fileA) and then do a comparison against fileB, outputting the different lines to fileC.

Note that the text to remove always follows specific patterns. So I can find the text using grep with those patterns, but I don't know how to remove it...

Here is an example of the two files to show what I mean:

  • fileB

    qaqa rara
    abc 10:12:25 08/20/2014 123456 def
    ghi fff ddd
    jkl 09:20:40 08/20/2014 978645 dfdf gggg
  • fileA

    qaqa rara
    abc 10:32:15 07/15/2014 121456 xxx
    ghi eee ddd
    jkl 10:01:22 07/15/2014 971645 dfdf gggg

I want to find the difference between the above two files, disregarding the time (e.g 10:12:25), date (e.g. 08/20/2014) or id code (e.g. 123456) and output the differences to fileC

The two lines which are different, therefore are lines 2 and 3. Line 1 is the same for both files. Line 4 is the same for both files when the time, date and id info has been removed.

  • 1
    Please give us example input. – cuonglm Aug 20 '14 at 10:15

If your timestamps are consistently formated, you could strip them off (with sed, for example) before processing the files with whatever differencing method, e.g.

diff <(sed -E 's|[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{2,4} [0-9]{1,} ||' fileA) <(sed -E 's|[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{2,4} [0-9]{1,} ||' fileB)

Testing on your supplied input files:

$ diff \
<(sed -E 's|[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{2,4} [0-9]{1,} ||' fileA) \
<(sed -E 's|[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{2,4} [0-9]{1,} ||' fileB)
< abc xxx
< ghi eee ddd
> abc def
> ghi fff ddd
  • I tried that and get the error message "Missing name for redirect". – didjek Aug 20 '14 at 12:29
  • Apologies - my cut'n'paste had included additional shell prompt characters > at the start of the lines - please try with the corrected version above – steeldriver Aug 20 '14 at 12:45
  • I tried it but I still see the same "Missing name for redirect" message. – didjek Aug 20 '14 at 13:50
  • What is your OS and shell? I have broken out the actual command as a one-liner in case it is the continuation that is causing problems – steeldriver Aug 20 '14 at 13:53
  • Red Hat 6.4 and csh; I changed to bash and your suggestion worked, thanks! I will check it on the original file and mark it as solved if it works on that. – didjek Aug 20 '14 at 14:01
diff \
<(sed -r 's\[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{4} [0-9]{6} \\' fileA) \
<(sed -r 's\[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{2}/[0-9]{2}/[0-9]{4} [0-9]{6} \\' fileB) \
| egrep '^> ' | sed -r 's/^> //' > fileC


Strip the irrelevant portions as given in the OP's question from fileA and fileB and feed this into diff.

diff will output the changed sections with a preceeding "> ", so ignore everything else except the changes.

Finally strip the leading "> " from the output and store that in fileC as per the question.

I originally did it slightly differently but I just noticed that the files could vary on the irrelevant section so it has to be pre-stripped not post-stripped else diff would output info that hasn't actually changed when considering the relevant portions only.

Given the example input, cat fileC gives:

abc def
ghi fff ddd

The sed command is searching for the supplied regular expression which describes the irrelevant data, and replacing that with an empty string - ie, it deletes it.

  • I'm unsure how to use the first answer you provided. Here is what I tried to check for differences regardless of the time value: codediff newfile.txt otherfile.txt | egrep '^> [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}' | sed 's/^> //'code - however it did not produce any result, so I guess I'm doing something wrong. T – didjek Aug 20 '14 at 11:53
  • Now that you gave an example, I've put in the exact code you need above. – Ian Macintosh Aug 20 '14 at 13:36
  • @IanMacintosh - you can break down what your sed is doing a bit, you don't have to do an entire recap of the man pages but braiam's right. We like the A's here to be self contained explanations, within reason 8-). – slm Aug 20 '14 at 13:50
  • Good point @slm. I'll make the answer a bit more verbose. – Ian Macintosh Aug 20 '14 at 13:53

The easiest command to get result would be the below

$ diff <(tr -s "[0-9],:,/" " " < fileA) <(tr -s "[0-9],:,/" " " < fileB)

The command is very straight forward and there is no complex regular expression as well.

Sample output will be as below

< abc xxx
< ghi eee ddd
> abc def
> ghi fff ddd

Hope this is what you want.

{   paste -d\| /dev/fd/3 /dev/fd/4 |
    sed '/\([^ ]*\) [0-9:/ ]*\(.*\)|\1 .*\2/d;=' |
    sed 'N;s/\(\n\)\(.*\)|/:\tFILEA: \2\1\tFILEB: /'
} 3<<\FILEA 4<<\FILEB
qaqa rara
abc 10:12:25 08/20/2014 123456 def
ghi fff ddd
jkl 09:20:40 08/20/2014 978645 dfdf gggg
qaqa rara
abc 10:32:15 07/15/2014 121456 xxx
ghi eee ddd
jkl 10:01:22 07/15/2014 971645 dfdf gggg


2:      FILEA: abc 10:12:25 08/20/2014 123456 def
        FILEB: abc 10:32:15 07/15/2014 121456 xxx
3:      FILEA: ghi fff ddd
        FILEB: ghi eee ddd

You needn't be rid of the time and date - they're no great hurdle so long as the characters that compose them are reliable.

In the above pipeline paste first appends the corresponding line from FILEB to the tail of each of FILEA's lines with a single | separator then prints the results to stdout.

sed picks up the stream and compares:

  • the first sequence of 0 or more characters that are not space (referenced as \1)

  • all characters that occur between the following sequences : (referenced as \2)

    • at least a single <space> character then 0 or more of any of the following:

    • <space> characters

    • <digit> characters

    • <:colon> characters

    • </slash> characters

  • up to but not including the last occurring | character on the line

...with |\1.*\2. If they match sed deletes the line. If not it prints the line preceded by its line number.

The final sed process just pretties the output (I hope).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.