1

I would like to split an input file on character count (ASCII is fine), combined with new lines as well. That is, every group of 10000 character should be seen as one record to be piped into the child process, but if that 10000th character does not happen to be at the end of line, the whole line should be included (and thus more that 10000 characters are provided). Each line should be considered as a single entity, which cannot be split.

Is that possible with GNU parallel (or possibly with a chain of other tools which might be useful)?

3
  • In that case you could use split man7.org/linux/man-pages/man1/split.1.html with -b flag
    – Inian
    Commented Jul 30, 2021 at 12:52
  • Interesting... how can that be combined with parallel to keep processing the chunks in parallel? Should I introduce an artificial boundary character to allow parallel to split on? Commented Jul 30, 2021 at 13:26
  • With regards to split, I also think it should be split -C 10k, and not -b. Is that correct? Commented Jul 30, 2021 at 14:06

1 Answer 1

0

What you are asking for is pretty much:

seq 100000 | parallel --block 10k --pipe wc

It will pass a block around 10000 bytes to wc but will only give full lines.

It will not guarantee that the block will be at least 10 kbytes, but it will at most be one line off.

1
  • Great! The guarantees are fine for my case. Commented Aug 2, 2021 at 20:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .