3

I try to use the AmazonCloud-Storage using acd_cli to put my media files into the cloud. Unfortunately acd_cli don't work with large files. Therefore I'm search for a (virtual) filesystem which split large files into configurable peaces on the storage side.

E.g. mount.splitfs If the user stores a large file into the the data is only written into but files exceeding the configured size will be stored as file.{1..n}.

The complete setup will be like:

unionfs --+-- local_encfs (RW) ----- local_splitfs ----- local_storage
          |
          +-- acd_encfs (RO) ----- acd_splitfs ----- acd_cli_storage

The local and acd path is necesarry as acd_cli only support writing a file in sequence but encfs writes in random positions. New files will be written localy and a cron job moves them from the local_storage into the acd_cli_storage if they are older than x days.

I got most of all working but the splitfs is missing. Any ideas how the realize the splitfs?

Kind regards, Rüdiger

  • glusterfs? (Not sure if it can do that, and even if it does I don't know if it's appropriate for your use case) – Gilles Oct 14 '16 at 22:12
0

chunkfs.

ChunkFS is a FUSE based filesystem that allows you to mount an arbitrary file or block device as a directory tree of files that each represent a chunk of user-specified size of the mounted file.

More info at http://www.linux-mag.com/id/7437/

  • Chunkfs devides the complete fs into chunks. The critical size of the amazon cloud drive is 10GiB. I have to store 20TB. This causes the creation of > 2000 chunk files and mount them together. But in the 20TB are much less than this count of files which must be spitt into 2 ore 3 pieces. – Rüdiger Gubler Oct 15 '16 at 9:05
  • I tested with some localy created files (dd, losetup, mkfs, mount, unionfs) like describet in the referenced articel. Unfortunately I can't create a file larger than one chunk in the union. – Rüdiger Gubler Oct 15 '16 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.