How to split a file into paragraphs and name the resulting pieces based on an identifier present in each paragraph

Question

I have a big file with more than 3264880 lines. I wanted to split that file based on Two strings "BEGIN JOB" and " END JOB" and write it into multiple files and the file name should be based on a certain Identifier which is present between the lines BEGIN JOB and END JOB

Sample data

BEGIN JOB
  Identifier "ADHOC_Extract"
   DateModified "2018-10-02"
   TimeModified "15.09.52"
   BEGIN DSRECORD
      Identifier "ROOT"
      OLEType "CJobDefn"
      Readonly "0"
      Name "ADHOC_Extract"
END JOB

BEGIN JOB
  Identifier "HOC_Extract"
   DateModified "2018-11-02"
   TimeModified "12.09.52"
   BEGIN DSRECORD
      Identifier "ROOT"
      OLEType "CJobDefn"
      Readonly "0"
      Name "HOC_Extract"
END JOB

Output expected as two files since my sample has just two... But it will have more than 1000 such repeated patterns

ADHOC_Extract.txt

BEGIN JOB
  Identifier "ADHOC_Extract"
   DateModified "2018-10-02"
   TimeModified "15.09.52"
   BEGIN DSRECORD
      Identifier "ROOT"
      OLEType "CJobDefn"
      Readonly "0"
      Name "ADHOC_Extract"
END JOB

 HOC_Extract.txt

BEGIN JOB
  Identifier "HOC_Extract"
   DateModified "2018-11-02"
   TimeModified "12.09.52"
   BEGIN DSRECORD
      Identifier "ROOT"
      OLEType "CJobDefn"
      Readonly "0"
      Name "HOC_Extract"
END JOB

I am ok even to write a shell script for the same

Answer 1

With GNU awk

gawk -v RS="" '
    match($0, /Identifier "([^"]+)/, m) {
        print > (m[1]".txt")
        close(m[1]".txt")
    }
' sample.txt

With Perl, using the convenience Path::Tiny module from CPAN

perl -MPath::Tiny -00 -ne '/Identifier "(.+?)"/ and path("$1.txt")->spew($_)' sample.txt 

Answer 2

If the paragraphs in the data have the same format (i.e. ten lines per paragraph), then the command split is very effective (split man).

#!/bin/bash   
#remove blank lines from the original dataset.

awk NF original_data.txt > Free_spaces_data.txt 

# split the dataset into files (paragraph per file), each paragraph is 10 lines.

split -l 10 Free_spaces_data.txt new 

 #rename the files based on the internal name within each paragraph

for f in ./new*?; do
name=$(cat $f | awk -F'"' '/Name/{print $2}')
  mv "${f}" "${name}.txt"; 
done

Answer 3

This will take the first "Identifier" line to extract the file name:

awk '
!/^ *$/         {BUF = BUF RS $0
                }

! FN  &&
/Identifier/    {FN = $NF ".txt"
                 gsub (/"/, "", FN)
                }
/END JOB/       {print BUF > FN
                 BUF = FN = ""
                }
' file

It skips empty lines, appends the actual line to a buffer, creates a file name on first (FN empty) "Identifier" occurrence, removing any ", and prints the buffer on END JOB, resetting BUF and FN to empty.