3

I'd like to split the contents of a .txt file into multiple files, but I'm encountering two questions about limitations of csplit:

(1) can anyone offer a way around csplit's maximum limit of '99' file splits? I have a file with up to 384 splits based on a recurring blank line or character. I'd like csplit to be able to accomodate this with {*}, but this exceeds csplit's intrinsic file generation capacity.

(2) does anyone know of a way to pass the contents of a file to csplit (pipe to csplit), or can csplit only be used in its conventional way of calling a file in place? i.e. csplit -f split_name file_to_split.txt /split/ {*} vs. [series of commands] | csplit -f split_name /split/ {*}

Thank you for any suggestions, or alternatives to accomplish a similar task.

Improve this question
4
  • 1
    What csplit implementation do you have? Does it work if you supply a longer suffix format explicitly e.g. -b '%03d'?
    – steeldriver
    Mar 25, 2016 at 22:47
  • Good questions...I must not have gnu csplit (Mac OS 10.9.5), because csplit in my command line doesn't accept pipes and has the 99-piece limit. How can gnu csplit be implemented? @steeldriver, what is meant by supplying a longer suffix format explicitly?
    – kehmsen
    Mar 25, 2016 at 22:52
  • OSX csplit reads from stdin too, check your man page: If file is a dash ('-'), csplit reads from standard input. You can get gnu coreutils on mac via homebrew, google for more details.
    – don_crissti
    Mar 25, 2016 at 22:55
  • 1
    BSD csplit doesn't have {*} at all, but it does have the POSIX -n option, Use number decimal digits to form filenames for the file pieces, which I just tested and it seems to work when you give the appropriate repetition count.
    – Michael Homer
    Mar 25, 2016 at 22:56

1 Answer 1

Reset to default
8

As Michael Homer remarks, any POSIX-compliant csplit can take more than 99 files, but you have to pass the -n option.

csplit -n 3 file_to_split.txt

This being said, if you have a task that can be done easily with the GNU version of csplit but not with the FreeBSD/OSX version, you can install the GNU version (part of GNU coreutils). GNU coreutils can be installed through several sources, I think the most popular is homebrew. See also How to replace Mac OS X utilities with GNU core utilities?.

Alternatively, most of what csplit does can be implemented in a few lines of awk, and that gives you more flexibility in terms of things like the output file name. The following snippet reads from standard input and writes lines to part-000.txt, part-001.txt, etc., changing files every time a line contains header-line.

awk '
    /header-line/ {
        ++part;
        if (output_file) close(output_file);
        output_file=sprintf("part-%03d.txt", part)
    }
    {print >output_file}
'
Improve this answer
4
  • Gets me this after 17 files already: awk: part-017.txt makes too many open files - but I'm using awk on Mac, maybe it's different...
    – Bernhard
    Jul 3, 2018 at 10:24
  • 1
    @Bernhard No, that was my bad, I wasn't closing the file. I fixed the code.
    – Gilles 'SO- stop being evil'
    Jul 3, 2018 at 10:39
  • Awk doesn't have a snprintf function and I can't find any documentation for any implementation that does. Running this outputs: awk: calling undefined function snprintf input record number 1, file source line number 5 when I run this on a Mac.
    – Starfish
    Feb 26 at 19:00
  • @Starfish Sorry, typo, that's sprintf
    – Gilles 'SO- stop being evil'
    Feb 26 at 19:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .