I have a file like this :

1   Record|1111|ABC
2   text in between for record 1
3   text in between for record 1
4   Record|2222|XYZ
5   text in between for record 2    
6   Record|3333|XYZ
7   text in between for record 3
8   .

I want to read this file and generate something like

<Record_number> | <start line> | <number of lines> | md5sum(content)

That is:

1111|1|2|md5sum(Record|1111|ABC\ntext in between for record 1\ntext in between for record 1)
2222|4|1|md5sum(Record|2222|XYZ\ntext in between for record 2\n)


Currently, I am doing this using a two step process:

Step 1:

grep -n -C 0 "Record|" ../test.txt | awk -F[':|'] '{print $3"|"$1}'

will create


Step 2: Read this file line by line and generate md5sum and number of lines through script.

The issue it this two step processing is taking more processing time and the file size is huge (~4GB).

Is there a better way to do this?

  • is the file really has a line number? – Avinash Raj Nov 21 '14 at 9:34
  • No. It doesnt have. I gave it for understanding.. – Sinoop Joy Nov 21 '14 at 10:55
  • it's a tough job. did you wanna python solution? – Avinash Raj Nov 21 '14 at 11:54
  • Perl or unix is i am mainly looking for. – Sinoop Joy Nov 21 '14 at 12:01
  • python is preinstalled on most of the linux distributions. I don't know what's wrong with the dangerous but user friendly python language. – Avinash Raj Nov 21 '14 at 12:05

Based on Costas' answer.

1) Create a file parse.awk, with the following content :

/^Record/ {
  if (s>0) {
    printf ("%s|%s|", r,l)
    system("echo '"line"' | md5sum - | awk '{print $1}' ");
!/^Record/ {
  printf ("%s|%s|", r,l)
  system("echo '"line"' | md5sum - | awk '{print $1}' ");

See Costas' explanations. This script just do printf the start of the resulting line (rather than print, that puts a newline) system(echo $line | md5sum) to print the md5 - and a newline

2) Run awk -F"|" -f parse.awk myfile

3) Enjoy the result :

  • line=line"§"$0; ? is "§" equivalent to "\n" – Sinoop Joy Nov 21 '14 at 13:25
  • Sorry, there should be "\n" rather than § (I used § for testing purposes). Corrected – Pierre-Olivier Vares Nov 21 '14 at 13:26
  • This fails if we have Record in any form in in between lines. Eg : "text in between for record 2(Record)" – Sinoop Joy Nov 24 '14 at 10:32
  • Corrected the above issues by adding ^ – Sinoop Joy Nov 25 '14 at 6:59

Mostly it can be

awk -F"|" -v OFS="|" '
function md5(lines){
  func="printf \"%s\" \""lines"\"|md5sum|cut -f1 -d\ " ;
  func | getline v;
  return v
    print r,l,c,md5(line);
  print r,l,c,md5(line);
}' file

Briefly code explanation:

  1. Change field separators (input and output) to |

  2. Assign md5 function (thanks to Pierre-Olivier Vares for idea) to calculate md5sum for apropriate input lines. (may be there is better way to do it - you are welcome to comment)

  3. For lines which have Record word put necessary fields into variables and break counters to 1 and prints previous formatted line starting from 2nd occurence of Record word (for 2nd prints 1st, for 3rd prints 2nd and so on).

  4. For lines which have not Record word just add oneself to line variable and add 1 to counter c

  5. When finish prints the last formatted line (because last line is stored in memory and should be printed when will meet next Record word but have met end of file)

  • Can you explain the code ? :-) I am very new to unix. – Sinoop Joy Nov 21 '14 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.