2

I'd like to split the contents of a .txt file into multiple files, but I'm encountering two questions about limitations of csplit:

(1) can anyone offer a way around csplit's maximum limit of '99' file splits? I have a file with up to 384 splits based on a recurring blank line or character. I'd like csplit to be able to accomodate this with {*}, but this exceeds csplit's intrinsic file generation capacity.

(2) does anyone know of a way to pass the contents of a file to csplit (pipe to csplit), or can csplit only be used in its conventional way of calling a file in place? i.e. csplit -f split_name file_to_split.txt /split/ {*} vs. [series of commands] | csplit -f split_name /split/ {*}

Thank you for any suggestions, or alternatives to accomplish a similar task.

  • 1
    What csplit implementation do you have? Does it work if you supply a longer suffix format explicitly e.g. -b '%03d'? – steeldriver Mar 25 '16 at 22:47
  • Good questions...I must not have gnu csplit (Mac OS 10.9.5), because csplit in my command line doesn't accept pipes and has the 99-piece limit. How can gnu csplit be implemented? @steeldriver, what is meant by supplying a longer suffix format explicitly? – kehmsen Mar 25 '16 at 22:52
  • OSX csplit reads from stdin too, check your man page: If file is a dash ('-'), csplit reads from standard input. You can get gnu coreutils on mac via homebrew, google for more details. – don_crissti Mar 25 '16 at 22:55
  • BSD csplit doesn't have {*} at all, but it does have the POSIX -n option, Use number decimal digits to form filenames for the file pieces, which I just tested and it seems to work when you give the appropriate repetition count. – Michael Homer Mar 25 '16 at 22:56
5

As Michael Homer remarks, any POSIX-compliant csplit can take more than 99 files, but you have to pass the -n option.

csplit -n 3 file_to_split.txt

This being said, if you have a task that can be done easily with the GNU version of csplit but not with the FreeBSD/OSX version, you can install the GNU version (part of GNU coreutils). GNU coreutils can be installed through several sources, I think the most popular is homebrew. See also How to replace Mac OS X utilities with GNU core utilities?.

Alternatively, most of what csplit does can be implemented in a few lines of awk, and that gives you more flexibility in terms of things like the output file name. The following snippet reads from standard input and writes lines to part-000.txt, part-001.txt, etc., changing files every time a line contains header-line.

awk '
    /header-line/ {
        ++part;
        if (output_file) close(output_file);
        output_file=snprintf("part-%03d.txt", part)
    }
    {print >file}
'
  • Gets me this after 17 files already: awk: part-017.txt makes too many open files - but I'm using awk on Mac, maybe it's different... – Bernhard Jul 3 '18 at 10:24
  • 1
    @Bernhard No, that was my bad, I wasn't closing the file. I fixed the code. – Gilles 'SO- stop being evil' Jul 3 '18 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.