0

I have a big file with more than 3264880 lines. I wanted to split that file based on Two strings "BEGIN JOB" and " END JOB" and write it into multiple files and the file name should be based on a certain Identifier which is present between the lines BEGIN JOB and END JOB

Sample data

BEGIN JOB
  Identifier "ADHOC_Extract"
   DateModified "2018-10-02"
   TimeModified "15.09.52"
   BEGIN DSRECORD
      Identifier "ROOT"
      OLEType "CJobDefn"
      Readonly "0"
      Name "ADHOC_Extract"
END JOB

BEGIN JOB
  Identifier "HOC_Extract"
   DateModified "2018-11-02"
   TimeModified "12.09.52"
   BEGIN DSRECORD
      Identifier "ROOT"
      OLEType "CJobDefn"
      Readonly "0"
      Name "HOC_Extract"
END JOB

Output expected as two files since my sample has just two... But it will have more than 1000 such repeated patterns

ADHOC_Extract.txt

BEGIN JOB
  Identifier "ADHOC_Extract"
   DateModified "2018-10-02"
   TimeModified "15.09.52"
   BEGIN DSRECORD
      Identifier "ROOT"
      OLEType "CJobDefn"
      Readonly "0"
      Name "ADHOC_Extract"
END JOB

 HOC_Extract.txt

BEGIN JOB
  Identifier "HOC_Extract"
   DateModified "2018-11-02"
   TimeModified "12.09.52"
   BEGIN DSRECORD
      Identifier "ROOT"
      OLEType "CJobDefn"
      Readonly "0"
      Name "HOC_Extract"
END JOB

I am ok even to write a shell script for the same

1
  • There are two "Identifiers" in each JOB - what to do?
    – RudiC
    Commented Oct 12, 2018 at 22:04

3 Answers 3

4

With GNU awk

gawk -v RS="" '
    match($0, /Identifier "([^"]+)/, m) {
        print > (m[1]".txt")
        close(m[1]".txt")
    }
' sample.txt

With Perl, using the convenience Path::Tiny module from CPAN

perl -MPath::Tiny -00 -ne '/Identifier "(.+?)"/ and path("$1.txt")->spew($_)' sample.txt 
1
  • If there are multiple jobs with the same nome, only the last such job would be saved. It's an easy fix though, just change > to >> and make sure that the output files don't exists before running the awk script.
    – Kusalananda
    Commented Oct 13, 2018 at 8:25
0

If the paragraphs in the data have the same format (i.e. ten lines per paragraph), then the command split is very effective (split man).

#!/bin/bash   
#remove blank lines from the original dataset.

awk NF original_data.txt > Free_spaces_data.txt 

# split the dataset into files (paragraph per file), each paragraph is 10 lines.

split -l 10 Free_spaces_data.txt new 

 #rename the files based on the internal name within each paragraph

for f in ./new*?; do
name=$(cat $f | awk -F'"' '/Name/{print $2}')
  mv "${f}" "${name}.txt"; 
done
0

This will take the first "Identifier" line to extract the file name:

awk '
!/^ *$/         {BUF = BUF RS $0
                }

! FN  &&
/Identifier/    {FN = $NF ".txt"
                 gsub (/"/, "", FN)
                }
/END JOB/       {print BUF > FN
                 BUF = FN = ""
                }
' file

It skips empty lines, appends the actual line to a buffer, creates a file name on first (FN empty) "Identifier" occurrence, removing any ", and prints the buffer on END JOB, resetting BUF and FN to empty.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .