0

This question is similar to Splitting text files based on a regular expression, but not quite the same. My problem is that I don't want to split it into a specific number of files, but I actually want to split it based on the number of matches. For example: I have a 457 MB file and trying to split it down into much smaller files. Here's what's currently working:

csplit -z Scan.nessus /\<ReportHost/ '{*}'

However, this creates about 61.5k files for me as I have a ton of these <ReportHost entries in this 457MB file. Ultimately, I'd like to break this down by every 50 entries rather than every single entry.

Is there a way to modify this to accomplish that? I tried doing this in Ruby to some extent, but it seems to max out the VM's memory trying to parse through the file with Nokogiri.

2

1 Answer 1

1

Reusing the awk-approach

BEGIN{outfile="part_0" ; j=-1}
/\<ReportHost/ {j++}
j==50 {outfile="part_"++i ; j=0}
{print > outfile}

Run as

awk -f split.awk Scan.nessus

Problem of not removing the old file while processing remains.

For zero-padded outfiles, you may use

outfile="part_"sprintf("%04d",i++)

and "part_0000" in the BEGIN block. Then hope not to have more than 10,000 files. Or check the number of files you expect beforehand.

2
  • Hmm unfortunately this provides the following error: awk: split.awk: line 3: syntax error at or near 1. Tried fixing this myself but not familiar enough with awk to this extent. Dec 12, 2021 at 16:09
  • @user3447014 should have been ++i ... i is the output file index that is to be increased.
    – FelixJN
    Dec 12, 2021 at 16:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .