This question already has an answer here:

I have a large file that's a couple hundred lines long. This file is partitioned into many parts by a specific identifier, lets say 'ABC'. This line 'ABC' appears 6 times so I want 6 output files. I'm familiar with split and awk but can't seem to create a command line that will do what I've described, any ideas?

Here's an example

line 1
line 2
line 3
line 1
line 2

I'd like three files where ABC is the first line in the new file and it ends before the next ABC is encountered.

marked as duplicate by don_crissti, Jakuje, Jeff Schaller, Scott, Anthon Feb 17 '16 at 20:11

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    csplit is usually good for this kind of thing - however not knowing exactly what you mean by "partitioned by" and whether you want the ABCs to be part of the output it's hard to suggest a specific command – steeldriver Feb 17 '16 at 18:42
  • @steeldriver, I've added some clarifications. will look into csplit – openingceremony Feb 17 '16 at 18:56

Using csplit

csplit -z somefile /ABC/ {*}

The output files will be xx00, xx01, ... by default but you can change the format and numbering if desired - see man csplit

csplit -f splitfile_ $HAYSTACK /$NEEDLE/ "{$(($(grep -c $NEEDLE HAYSTACK)-1))}"
for file in splitfile_*; do
    sed --in-place "s/$NEEDLE//" $file

The above will split the file as requested no matter how many instances of the marker line you have, and then remove the marker from the resultant files. The output files will be called e. g. splitfile_00, splitfile_01, and so forth.

Picking apart that bit at the end of the csplit invocation: "{$(($(grep -c $NEEDLE HAYSTACK)-1))}": We use the subshell grep to get the number of instances of your marker within the file, and subtract one- this tells csplit just exactly how many splits it's going to be making.

Note that as written, things might go pear-shaped if your marker appears within the data.

Not the answer you're looking for? Browse other questions tagged or ask your own question.