Read file and find all occurrences and generate hash for the content between the occurrences

Question

I have a file like this :

1   Record|1111|ABC
2   text in between for record 1
3   text in between for record 1
4   Record|2222|XYZ
5   text in between for record 2    
6   Record|3333|XYZ
7   text in between for record 3
8   .

I want to read this file and generate something like

<Record_number> | <start line> | <number of lines> | md5sum(content)

That is:

1111|1|2|md5sum(Record|1111|ABC\ntext in between for record 1\ntext in between for record 1)
2222|4|1|md5sum(Record|2222|XYZ\ntext in between for record 2\n)

etc.

Currently, I am doing this using a two step process:

Step 1:

grep -n -C 0 "Record|" ../test.txt | awk -F[':|'] '{print $3"|"$1}'

will create

1111|1
2222|4
3333|6

Step 2: Read this file line by line and generate md5sum and number of lines through script.

The issue it this two step processing is taking more processing time and the file size is huge (~4GB).

Is there a better way to do this?

Answer 1

Based on Costas' answer.

1) Create a file parse.awk, with the following content :

/^Record/ {
  if (s>0) {
    printf ("%s|%s|", r,l)
    system("echo '"line"' | md5sum - | awk '{print $1}' ");
  }
  s=1;
  r=$2;
  c=1;
  l=NR;
  line="$0";
}
!/^Record/ {
  line=line"\n""$0";
  c+=1
}
END {
  printf ("%s|%s|", r,l)
  system("echo '"line"' | md5sum - | awk '{print $1}' ");
}

See Costas' explanations. This script just do printf the start of the resulting line (rather than print, that puts a newline) system(echo $line | md5sum) to print the md5 - and a newline

2) Run awk -F"|" -f parse.awk myfile

3) Enjoy the result :

1111|1|cb36533781d8dd00011a85b0db9b87b3
2222|4|521331bb249e8a668afa2199fa8d289a
3333|6|6c2564464187094e9db3159d26ade2a5

Answer 2

Mostly it can be

awk -F"|" -v OFS="|" '
function md5(lines){
  func="printf \"%s\" \""lines"\"|md5sum|cut -f1 -d\ " ;
  func | getline v;
  return v
}
/Record/{
  if(s>0)
    print r,l,c,md5(line);
  s=1;
  r=$2;
  c=1;
  l=NR;
  line=$0
}
!/Record/{
  line=line"\n"$0;
  c+=1
}
END{
  print r,l,c,md5(line);
}' file

Briefly code explanation:

1. Change field separators (input and output) to |

2. Assign md5 function (thanks to Pierre-Olivier Vares for idea) to calculate md5sum for apropriate input lines. (may be there is better way to do it - you are welcome to comment)

3. For lines which have Record word put necessary fields into variables and break counters to 1 and prints previous formatted line starting from 2nd occurence of Record word (for 2nd prints 1st, for 3rd prints 2nd and so on).

4. For lines which have not Record word just add oneself to line variable and add 1 to counter c

5. When finish prints the last formatted line (because last line is stored in memory and should be printed when will meet next Record word but have met end of file)