1

I would like to split an input file on character count (ASCII is fine), combined with new lines as well. That is, every group of 10000 character should be seen as one record to be piped into the child process, but if that 10000th character does not happen to be at the end of line, the whole line should be included (and thus more that 10000 characters are provided). Each line should be considered as a single entity, which cannot be split.

Is that possible with GNU parallel (or possibly with a chain of other tools which might be useful)?

Improve this question
3
  • In that case you could use split man7.org/linux/man-pages/man1/split.1.html with -b flag
    – Inian
    Jul 30, 2021 at 12:52
  • Interesting... how can that be combined with parallel to keep processing the chunks in parallel? Should I introduce an artificial boundary character to allow parallel to split on?
    – Frazier Thien
    Jul 30, 2021 at 13:26
  • With regards to split, I also think it should be split -C 10k, and not -b. Is that correct?
    – Frazier Thien
    Jul 30, 2021 at 14:06

1 Answer 1

Reset to default
0

What you are asking for is pretty much:

seq 100000 | parallel --block 10k --pipe wc

It will pass a block around 10000 bytes to wc but will only give full lines.

It will not guarantee that the block will be at least 10 kbytes, but it will at most be one line off.

Improve this answer
1
  • Great! The guarantees are fine for my case.
    – Frazier Thien
    Aug 2, 2021 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.