I'm thinking a way of storing large files on vfat file system. Obviously, the only way to store files larger than 4GiB is to split them. I'm aware that we can use split and cat commands for splitting and merging files. However, to use those commands, more space is needed for resulting file(s). If I'm understanding how file systems work correctly, no file is actually deleted and deallocated until they're closed. To use as little space as possible, it would take funky technique that involves truncate() and reversing the file content(reading and shrinking the original file as it goes).

Is there any kernel module that creates a loop device out of multiple split files? Or a util command that does the exact same idea as mine?

Or I'll start making one of them. Oh! Can I do that with mdadm? The linear level?

  • btw, I know how to do things like cat file.tar.00 file.tar.01 file.tar.02 | tar xf - – Ashe the human Feb 2 '17 at 0:52
  • Then what is your practical use that you can't do by yourself? Do note that truncating a file as you read it would imply rewriting the whole remains of the file after every segment you read, thus extremely slow. – Julie Pelletier Feb 2 '17 at 0:55
  • If the split files don't have odd sizes, you can do it loop devices and concatenate them with device mapper linear target. If they do have odd sizes, you need a user space block device or filesystem - fuse and nbd can be used for those. mdadm build linear might work too, never tried it. – frostschutz Feb 2 '17 at 1:55

While I dont understand your need to use a fat file system, this seems a duplicate of https://unix.stackexchange.com/a/17173/108069

In your case each of your dd files would be 4gb each.

  • OP mentioned a need to avoid using more disk space and there is absolutely no advantage doing things that way instead of using a pipe, quite the opposite. – Julie Pelletier Feb 2 '17 at 1:37
  • The advantage to loop mounting+mdadm over pipe is you dont need to create an entire second file somewhere, either in memory or on disk every time you want to access it. As long as it's mounted the entire file / contents are always available. – jammin Feb 6 '17 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.