2

I am trying to concatenate thousands of files that result from processing the pieces output by split with

find ./subdir/* -print0 | sort | xargs -0 cat > full_merge.txt

The files under subdir are:

something.foo.aaaaa.output
something.foo.aaaab.output
something.foo.aaaac.output

...

But I get File name too long

Why?

4

Not sure why you get that error.

find ./subdir/* -print0

(assuming --print0 was a typo) would output a big non-terminated line with a few NUL characters in it. Because it doesn't end in a newline character and because it contains NUL characters, it is not text, so the behaviour of text utilities like sort is unspecified there. GNU sort, would treat it as one big line and add back the missing newline.

Maybe you're using busybox sort which as far as I can tell treats those NUL characters on input as line separators as well but outputs lines terminated by newline characters.

xargs -0 expects NUL separated records. So with GNU sort, xargs would treat the sort output as the same list as generated by find -print0 plus on spurious element consisting of that extra newline character added by sort.

In the case of busybox sort, because all NULs have been converted to newlines, there's only one record as far as xargs -0 can tell, so it will call cat with one huge filename containing newline characters.

Here, if you want to sort the list of filenames, you need either to use GNU sort with its -z option:

find subdir -type f -print0 | sort -z | xargs -r0 cat > merge.txt

Or assume that filenames don't contain newline characters and run:

find subdir -type f | sort | tr '\n' '\0' | xargs -r0 cat > merge.txt

Since you used the zsh tag, you may as well do everything in zsh:

autoload zargs
zargs subdir/**/*(D.) -- cat > merge.txt

Which actually will provide you with a sorting order closer to what you're looking for.

2

xargs' -0 argument is for when the input is NUL-separated, which it's not. If you want this to work you have to separate with NULs in the preceding commands. Unfortunately there doesn't seem to be any way to make this POSIX compatible:

find ./subdir/* -print0 | sort -z | xargs -0 cat -- > full_merge.txt
4
  • 1
    Usually it's find -print0 | sort -z | xargs -0 May 28 '13 at 14:54
  • Yeah, I tried to make it as POSIX compatible as possible. Unfortunately POSIX xargs doesn't support -0 yet.
    – l0b0
    May 28 '13 at 14:57
  • 1
    And as per POSIX, sort is not required to handle non-text data. And -t is to specify the field separator, not the line/record separator. May 28 '13 at 16:16
  • Oops, corrected.
    – l0b0
    May 29 '13 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.