I've got 10k+ files totaling over 20GB that I need to concatenate into one file.
Is there a faster way than
cat input_file* >> out
The preferred way would be a bash command, Python is acceptable too if not considerably slower.
Nope, cat is surely the best way to do this. Why use python when there is a program already written in C for this purpose? However, you might however want to consider using
Allocating the space for the output file first may improve the overall speed as the system won't have to update the allocation for every write.
For instance, if on Linux:
Another benefit is that if there's not enough free space, the copy will not be attempted.
There's an API to generalise that to ref-copy all the files (the
If the source files are sparse or have large sequences of NUL characters, you could make a sparse output file (saving time and disk space) with (on GNU systems):