3

I found the following awk script in a previous answer for splitting large files into smaller parts based on a pattern (replacing "header-line" with a different pattern). However, I have the need to take it even further as there could be millions of parts created but if I can filter each one again it will get what I need.

awk '
    /header-line/ {++part}
    {print >sprintf("part-%03d.txt", part)}
'

Is it possible to interrogate each "part" either before or after it is written to check if a certain pattern exists in the part and only write that part out on a match for the additional pattern.

As an example, The initial criteria would be "ID: " (the header-line above). I then want to search each "part" created for a particular method name; call it "searchForThisMethod" as the string to search for which could be at various locations in the part.

Example Input:

ID: 1
bb
bb
bb
bb
secondaryCheck
ID: 2
b
b
b
b
b
b
b
b
ID: 3
h
h
h
h
h
secondaryCheck
g
g
g
g
6
  • can you give an example of such a pattern?
    – Jeff Schaller
    Commented Jun 27, 2016 at 18:08
  • Hi Jeff. The initial criteria would be "ID: " (the header-line above). I then want to search each "part" created for a particular method name; call it "searchForThisMethod" as the string to seartch for which could be at various locations in the part. Hope that makes sense.
    – user7972
    Commented Jun 27, 2016 at 18:11
  • 1
    You should edit your question to have these details; they can get overlooked in comments.
    – Jeff Schaller
    Commented Jun 27, 2016 at 18:24
  • Good idea. I have updated the question with that info.
    – user7972
    Commented Jun 27, 2016 at 18:27
  • Are the parts delimited in any way (for example, by a blank line)? Commented Jun 27, 2016 at 19:16

1 Answer 1

1

You could split the file then run grep -q some_method on each piece and delete it if false e.g.

for file in part-*.txt; do
grep -q some_method "$file"
  if [ $? -ne 0 ]
    then
      rm "$file"
  fi
done

or
use a two pass - removing from the input the "parts" that contain some_method then splitting the result e.g. using sed for the first pass and your awk code for the second pass:

sed -e '1{h;d;}' -e '/ID/!{H;$!d;}' -e 'x;/some_method/!d' infile | \
awk '/ID/{++part}{print > sprintf("part-%03d.txt", part)}'

sed copies the 1st line over the hold buffer and deletes it then appends each line that does not match ID to the Hold buffer, deletes it if it's not the last line and exchanges buffers on lines that match ID deleting the pattern space if it doesn't contain some_method. The result is then piped to awk. If you're getting too many open files error then you'll have to close() the files:

sed -e '1{h;d;}' -e '/ID/!{H;$!d;}' -e 'x;/some_method/!d' infile | \
awk '/ID/{++part}{close(fn);fn=sprintf("part-%03d.txt", part);print >> fn}'

or, if you're on gnu/anything, you could use csplit instead of awk:

sed '1{h;d};/ID/!{H;$!d};x;/some_method/!d' infile | \
csplit -f 'part-' -b '%03d.txt' -sz - /ID/ '{*}'
18
  • Thank Don. One question please, I want the result to be that I have the parts that match o the initital ("ID: ") and then the secondary. For the first option why would I delete if true?
    – user7972
    Commented Jun 27, 2016 at 21:49
  • Yes sir.. That is correct. First criteria is to split the file on "ID: " and then for those parts that are extracted, match on the secondary criteria (some_method). The end result being I have a list of "parts" that start with "ID: " and also have the "some"method" in the file.
    – user7972
    Commented Jun 27, 2016 at 21:53
  • @user7972 - no, it'll still be a two pass, as I said in my post; I've added the syntax for deleting the parts that don't contain the second pattern if you prefer splitting it first and then removing the unneeded pieces. The command assumes you're in the directory where you have the pieces. Commented Jun 27, 2016 at 22:09
  • Thank you. I'm going to test the sed/awk against my 10G input file. It should generate about individual 20k "parts". In conjunction, I am running the splits (of "ID:" against several other 10G files). But if the sed/awk works like I hope it is easier and less prone to errors as when running in 2 parts I first split the file itself into 10 1G parts as the filesystem was having a hard time keeping track of 2M individual files, Kind of a mess. But if the sed/awk works like I hope it will just run and spit out 20k little parts and I'm golden.
    – user7972
    Commented Jun 27, 2016 at 22:22
  • @user7972 - I've added two alternatives just in case you get a too many open files error with your huge input there... Commented Jun 27, 2016 at 23:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .