Create small record files from a big group of record file

Question

I have a file like this :

Mainfile:

Record|1111|ABC     
text in between for record 1     
text in between for record 1     
Record|2222|XYZ     
text in between for record 2     
Record|3333|XYZ     
text in between for record 3 

I want to create smaller files which contains each record data

File : 1111

Record|1111|ABC    
text in between for record 1    
text in between for record 1

File : 2222

Record|2222|XYZ
text in between for record 2

File : 3333

Record|3333|XYZ
text in between for record 3

Currently I am using awk to do this :

awk -F"|" '
/Record/{ 
        if(s>0) {
             print line > r;
         }
s=1; 
r=$2; 
line=$0 
} 
!/Record/
{ 
line=line"\n"$0; 
}

END
{
    print line > r;
}' file

But it is taking more time to process the file. Each file is 4GB in size and will contain around 75k+ Records.

Is there any faster way to do this ?

Answer 1

Using GNU awk, you can try:

gawk -v RS='Record[|]' -F'|' '$1{printf "%s%s", RT, $0 > $1}' file

With Record| as the record separator, we can easily obtain each record. GNU awk's RS can be a regular expression (as in this case), and RT contains the text which matched this expression (Record|).

With mawk, RT isn't available, so you can use printf "Record|%s", $0 instead.

Answer 2

Now if the need to base output file names on file content can be removed, coreutils csplit may be faster at this than awk

csplit file '/^Record[|]/' '{*}'

This will create a set of files xx01, xx02 and so on. You can ignore the xx01 file since in your case it is going to be empty. Also note that csplit lets you configure the prefix (xx) and suffix (01, 02....) parts of output file names