I've got 10k+ files totaling over 20GB that I need to concatenate into one file.
Is there a faster way than
cat input_file* >> out
?
The preferred way would be a bash command, Python is acceptable too if not considerably slower.
Nope, cat is surely the best way to do this. Why use python when there is a program already written in C for this purpose? However, you might want to consider using xargs in case the command line length exceeds ARG_MAX and you need more than one cat. Using GNU tools, this is equivalent to what you already have:
find . -maxdepth 1 -type f -name 'input_file*' -print0 |
sort -z |
xargs -0 cat -- >>out
-
1
-
1Yes, because the output of
findis piped throughsort. Without this, the files would be listed in an arbitrary order (defined by the file system, which could be file creation order). – scai Mar 5 '14 at 13:43 -
-
1@Kiwy, the only case I can see is if the locale isn't properly set in the environment, then sort might behave differently from a
bashglob. Otherwise I don't see any cases wherexargsorcatwould not behave as expected. – Graeme Mar 5 '14 at 14:05 -
3@MarcvanLeeuwen,
xargswill call as maycatas is necessary to avoid an E2BIG error of execve(2). – Stéphane Chazelas Mar 6 '14 at 9:41
Allocating the space for the output file first may improve the overall speed as the system won't have to update the allocation for every write.
For instance, if on Linux:
size=$({ find . -maxdepth 1 -type f -name 'input_file*' -printf '%s+'; echo 0;} | bc)
fallocate -l "$size" out &&
find . -maxdepth 1 -type f -name 'input_file*' -print0 |
sort -z | xargs -r0 cat 1<> out
Another benefit is that if there's not enough free space, the copy will not be attempted.
If on btrfs, you could copy --reflink=always the first file (which implies no data copy and would therefore be almost instantaneous), and append the rest. If there are 10000 files, that probably won't make much difference though unless the first file is very big.
There's an API to generalise that to ref-copy all the files (the BTRFS_IOC_CLONE_RANGE ioctl), but I could not find any utility exposing that API, so you'd have to do it in C (or python or other languages provided they can call arbitrary ioctls).
If the source files are sparse or have large sequences of NUL characters, you could make a sparse output file (saving time and disk space) with (on GNU systems):
find . -maxdepth 1 -type f -name 'input_file*' -print0 |
sort -z | xargs -r0 cat | cp --sparse=always /dev/stdin out
answered Mar 5 '14 at 14:57
-
1@XTian, no, it should be neither
>nor>>, but1<>as I said to write into the file. – Stéphane Chazelas Mar 5 '14 at 16:41 -
5@grebneke,
<>is the standard Bourne/POSIX read+write redirection operator. See your shell manual or the POSIX spec for details. The defaultfdis0for the<>operator (<>is short for0<>, like<is short for0<and>short for1>), so you need the1to explicitly redirect stdout. Here, it's not so much that we need read+write (O_RDWR), but that we don't wantO_TRUNC(as in>) which would deallocate what we've just allocated. – Stéphane Chazelas Mar 5 '14 at 20:30 -
1@grebneke, unix.stackexchange.com/search?q=user%3A22565+%22%3C%3E%22 will give you a few. ksh93 has seek operators BTW, and you can seek forward with
ddor via reading. – Stéphane Chazelas Mar 5 '14 at 21:15 -
1@StephaneChazelas - thanks a lot, your help and knowledge is deeply appreciated! – grebneke Mar 5 '14 at 21:19
-
1I'm not convinced that there will be many cases where
fallocatewill negate the overhead of the extrafind, even though it will be faster the second time round.btrfscertainly opens up some interesting possibilities though. – Graeme Mar 6 '14 at 1:28
finddoes not sort files the same as a shell glob. – Graeme Mar 5 '14 at 13:18outis located on another disk. – user55518 Mar 5 '14 at 22:41