Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a text file called entry.txt that contains the following:

[ entry1 ]
1239 1240 1242 1391 1392 1394 1486 1487 1489 1600
1601 1603 1657 1658 1660 2075 2076 2078 2322 2323
2325 2740 2741 2743 3082 3083 3085 3291 3292 3294
3481 3482 3484 3633 3634 3636 3690 3691 3693 3766
3767 3769 4526 4527 4529 4583 4584 4586 4773 4774
4776 5153 5154 5156 5628 5629 5631
[ entry2 ]
1239 1240 1242 1391 1392 1394 1486 1487 1489 1600
1601 1603 1657 1658 1660 2075 2076 2078 2322 2323
2325 2740 2741 2743 3082 3083 3085 3291 3292 3294
3481 3482 3484 3690 3691 3693 3766 3767 3769 4526
4527 4529 4583 4584 4586 4773 4774 4776 5153 5154
5156 5628 5629 5631
[ entry3 ]
1239 1240 1242 1391 1392 1394 1486 1487 1489 1600
1601 1603 1657 1658 1660 2075 2076 2078 2322 2323
2325 2740 2741 2743 3082 3083 3085 3291 3292 3294
3481 3482 3484 3690 3691 3693 3766 3767 3769 4241
4242 4244 4526 4527 4529 4583 4584 4586 4773 4774
4776 5153 5154 5156 5495 5496 5498 5628 5629 5631

I would like to split it into three text files: entry1.txt, entry2.txt, entry3.txt. Their contents are as follows.

entry1.txt:

[ entry1 ]
1239 1240 1242 1391 1392 1394 1486 1487 1489 1600
1601 1603 1657 1658 1660 2075 2076 2078 2322 2323
2325 2740 2741 2743 3082 3083 3085 3291 3292 3294
3481 3482 3484 3633 3634 3636 3690 3691 3693 3766
3767 3769 4526 4527 4529 4583 4584 4586 4773 4774
4776 5153 5154 5156 5628 5629 5631

entry2.txt:

[ entry2 ]
1239 1240 1242 1391 1392 1394 1486 1487 1489 1600
1601 1603 1657 1658 1660 2075 2076 2078 2322 2323
2325 2740 2741 2743 3082 3083 3085 3291 3292 3294
3481 3482 3484 3690 3691 3693 3766 3767 3769 4526
4527 4529 4583 4584 4586 4773 4774 4776 5153 5154
5156 5628 5629 5631

entry3.txt:

[ entry3 ]
1239 1240 1242 1391 1392 1394 1486 1487 1489 1600
1601 1603 1657 1658 1660 2075 2076 2078 2322 2323
2325 2740 2741 2743 3082 3083 3085 3291 3292 3294
3481 3482 3484 3690 3691 3693 3766 3767 3769 4241
4242 4244 4526 4527 4529 4583 4584 4586 4773 4774
4776 5153 5154 5156 5495 5496 5498 5628 5629 5631

In other words, the [ character indicates a new file should begin. The entries ([ entry*], where * is an integer) are always in numerical order and are consecutive integers starting from 1 to N (in my actual input file, N = 200001).

Is there any way I can accomplish automatic text file splitting in bash? My actual input entry.txt actually contains 200,001 entries.

share|improve this question

11 Answers 11

up vote 10 down vote accepted

And here's a nice, simple, gawk one-liner :

$ gawk '/^\[/{match($0, /^\[ (.+?) \]/, k)} {print >k[1]".txt" }' entry.txt

This will work for any file size, irrespective of the number of lines in each entry, as long as each entry header looks like [ blahblah blah blah ]. Notice the space just after the opening [ and just before the closing ].


EXPLANATION:

awk and gawk read an input file line by line. As each line is read, its contents are saved in the $0 variable. Here, we are telling gawk to match anything within square brackets, and save its match into the array k.

So, every time that regular expression is matched, that is, for every header in your file, k[1] will have the matched region of the line. Namely, "entry1", "entry2" or "entry3" or "entryN".

Finally, we print each line into a file called <whatever value k currently has>.txt, ie entry1.txt, entry2.txt ... entryN.txt.

This method will be much faster than perl for larger files.

share|improve this answer
    
+1 nice. You do not need to match the entry: /^\[/ { name=$2 } should be enough. –  Thor Aug 26 '12 at 1:51
    
Thanks @Thor. Your suggestion is correct for the described case, but it assumes there is never a space in the entry's name. That is why I used the example [ blahblah blah blah ] in my answer. –  terdon Aug 26 '12 at 13:20
    
Ah I missed the bit about space separated entries. You could also accommodate those with FS, e.g. -F '\\[ | \\]'. –  Thor Aug 26 '12 at 13:50
    
@terdon I really like this short solutions, unfortunately I usually fail to generalize them to my needs. Could you give me a hand? My file has lines starting with #S x, where x is a 1, 2, or 3 digit number. Just saving them into x.dat would suffice. I tried: gawk '/^#S/{match($0, / [0-9]* /, k)} {print >k[1]".dat" }' myFile.txt and some variations of that. –  mikuszefski 2 days ago
    
Got it gawk '/^#S/{match($0, /^#S (\s+?)([0-9]+)(\s+?)/, k)} {print >k[2]".txt" }' test.txt did the trick. Don't understand the array number 2 very well, though. –  mikuszefski yesterday

With csplit from GNU coreutils (non-embedded Linux, Cygwin):

csplit -f entry -b '%d.txt' entry.txt '/^\[ .* \]$/' '{*}'

You'll end up with an extra empty file entry0.txt (containing the part before the first header).

Standard csplit lacks the {*} indefinite repeater and the -b option to specify the suffix format, so on other systems you'll have to count the number of sections first and rename the output files afterwards.

csplit -f entry -n 9 entry.txt '/^\[ .* \]$/' "{$(egrep -c '^'\[ .* \]$' <entry.txt)}"
for x in entry?????????; do
  y=$((1$x - 1000000000))
  mv "entry$x" "entry$y.txt"
done
share|improve this answer
    
i find csplit is bit quirky once in a while, but incredibly useful when i want to do this sort of thing. –  ixtmixilix Aug 25 '12 at 23:59

Here's a short awk one-liner:

awk '/^\[/ {ofn=$2 ".txt"} ofn {print > ofn}' input.txt

How does this work?

  • /^\[/ matches lines starting with a left square bracket, and
  • {ofn=$2 ".txt"} sets a variable to the second white-spaced-delimited word as our output file name. Then,
  • ofn is a condition that evaluates to true if the variable is set (thus causing lines before your first header to be ignored)
  • {print > ofn} redirects the current line to the specified file.

Note that all of the spaces in this awk script can be removed, if compactness makes you happy.

share|improve this answer

In perl it can be done much simplier:

perl -ne 'open(F, ">", ($1).".txt") if /\[ (entry\d+) \]/; print F;' file
share|improve this answer

It can be done from the command line in python as:

paddy$ python3 -c 'out=0
> with open("entry.txt") as f: 
>   for line in f:
>     if line[0] == "[":
>       if out: out.close()
>       out = open(line.split()[1] + ".txt", "w")
>     else: out.write(line)'
share|improve this answer

This is a somewhat crude, but easily understood way to do it: use grep -l '[ entry ]' FILENAME to get the line numbers to split at [ entry ]. Use a combination off head and tail to get the right pieces.

Like I said; it isn't pretty, but is easy to comprehend.

share|improve this answer

What about using awk with [ as a record separator and space as the field separator. This gives us easily the data to be put in the file as $0 where he have to put back the removed leading [ and the filename as $1. We then only have to handle the special case of the 1st record which is empty. This gives us:

awk -v "RS=[" -F " " 'NF != 0 {print "[" $0 > $1}' entry.txt
share|improve this answer

terdon's answer works for me but I needed to use gawk, not awk. The gawk manual ( search for 'match(' ) explains that the array argument in match() is a gawk extension. Maybe it depends on your Linux install and your awk/nawk/gawk versions but on my Ubuntu machine only gawk ran terdon's excellent answer:

$ gawk '{if(match($0, /^\[ (.+?) \]/, k)){name=k[1]}} {print >name".txt" }' entry.txt
share|improve this answer

Here's a perl solution. This script detects the [ entryN ] lines and changes the output file accordingly, but doesn't validate, parse or process the data in each section, it just prints the input line to the output file.

#! /usr/bin/perl 

# default output file is /dev/null - i.e. dump any input before
# the first [ entryN ] line.

$outfile='/dev/null';
open(OUTFILE,">",$outfile) || die "couldn't open $outfile: $!";

while(<>) {
  # uncomment next two lines to optionally remove comments (starting with
  # '#') and skip blank lines.  Also removes leading and trailing
  # whitespace from each line.
  # s/#.*|^\s*|\s*$//g;
  # next if (/^$/)

  # if line begins with '[', extract the filename
  if (m/^\[/) {
    (undef,$outfile,undef) = split ;
    close(OUTFILE);
    open(OUTFILE,">","$outfile.txt") || die "couldn't open $outfile.txt: $!";
  } else {
    print OUTFILE;
  }
}
close(OUTFILE);
share|improve this answer

I prefer the csplit option but as an alternative here's an awk solution:

BEGIN { 
  RS="\\[ entry[0-9]+ \\]\n"  # Record separator
  ORS=""                      # Reduce whitespace on output
}
NR == 1 { f=RT }              # Entries are of-by-one relative to matched RS
NR  > 1 {
  split(f, a, " ")            # Assuming entries do not have spaces 
  print f  > a[2] ".txt"      # a[2] now holds the bare entry name
  print   >> a[2] ".txt"
  f = RT                      # Remember next entry name
}
share|improve this answer

Hi i wrote this simple script using ruby to solve your problem

#!ruby
# File Name: split.rb

fout = nil

while STDIN.gets
  line = $_
  if line.start_with? '['
    fout.close if fout
    fname = line.split(' ')[1] + '.txt'
    fout = File.new fname,'w'
  end
  fout.write line if fout
end

fout.close if fout

you can use it this way:

ruby split.rb < entry.txt

i have tested it, and it works fine..

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.