2

I have huge files {0..9}.bin which I want concatenate into out.bin. I don't need the original files afterwards. So I was wondering, if this is possible by only modifying the filesystem index without copying the file contents (see Append huge files to each other without copying them for efficient copy solutions).

On modern file systems (e.g. btrfs) cp --reflink=always exists. Fifos are on the file system level (at least btrfs send also tracks fifos), so they should have information about the actual data blocks used. Therefore, cp --reflink=always should be able to determine the extend numbers on disk and re-use them.

So I was wondering, if it is possible to use mkfifo in combination with cp --reflink=always?

Update Currently, it is not working:

for i in {1..9}; do dd if=/dev/urandom of="in$i.bin" bs=5M count=200; done;
mkfifo fifo
cat in* >fifo &
cp --reflink=always fifo out.bin

results in cp: failed to clone 'out.bin' from 'fifo': Invalid argument

Probably, it never will, since FIFOs have no information about the storage origin bug instead are just dumb pipes.

13
  • Is your intention to save space or to save on write operations?
    – Kusalananda
    May 20 '20 at 8:53
  • @Kusalanada: Save on write operations (speed).
    – darkdragon
    May 20 '20 at 10:26
  • Why do you think a fifo helps you achieve your goal? A fifo (named pipe) is a tool that enables pushing data into and out of a RAM buffer abstracted by a special file. It is not a file and does not map blocks.... Maybe you're on to something, I just can't work it out.
    – Pedro
    May 20 '20 at 10:26
  • @Pedro: Yes, I am new to fifos, but inspecting this code unix.stackexchange.com/a/80439/234981 leads me to the conclusion that cat *.bin > myfifo.bin does not actually copy the contents into RAM but instead reference them somehow.
    – darkdragon
    May 20 '20 at 10:28
  • 1
    @Pedro: I read this thread and actually have been inspired for my question after reading it! The use-case is the high-performance cloud storage provider minio. Currently (in fs mode), chunks are saved when they arrive for S3 compliance. After the last chunk has been received, all files are concatenated and and copied to the appropriate location. It is used by TBs of data every day!
    – darkdragon
    May 20 '20 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.