1

I'd like to do something along the lines of

mkfs -t btrfs filedrive
mount filedrive /media/fuse

Without specifying a particular size, I'd like to be able to have the file grow in size as I write files into the mounted filesystem, and shrink when files are deleted.

Is there some mechanism for this?


I have also seen this question and I am aware that it could in theory be managed manually, but my question has nothing to do with ecryptfs and focuses on the automatic aspect. I do not particularly care about the filesystem type inside the file.

I am also aware of existing systems that do something similar: specifically VirtualBox's Dynamic Disks. If there is some way of using those - or something akin to them - without actually running a virtual machine, I would also be happy with that.

  • That needs the filesystem driver operating on the underlying storage driver, seems to be impossible. Is this X-Y? – 炸鱼薯条德里克 Feb 19 at 4:11
  • @炸鱼薯条德里克 I've edited in the one instance I know of something similar. The question is more of a hypothetical, so isn't X-Y. – Jay Feb 19 at 5:28
  • 1
    Virtualization software should not be aware of any data structure stored on virtual disk. The Vbox functionality you mentioned is completely different from your description in the first paragraph. It's just acting like a normal disk controller but implemented by some magic power instead of real hardware storage. Even on a physical machine, the disk controller also doesn't care about data structures – 炸鱼薯条德里克 Feb 19 at 5:38
  • The functionality of mounting and changing filesystems inside of virtual disk image does exist, but is still different from what you described. AFAIK, filesystem is always a fix-sized data structure at some specific time point – 炸鱼薯条德里克 Feb 19 at 5:48
  • In both cases, a file is being used as a filesystem, so I do not see how it would be "completely different". – Jay Feb 19 at 6:17
6

Actually, something like it is already possible

A filesystem needs to have a defined maximum size. But as the filesystem-as-file can be sparse, that size can be an arbitrary number which doesn't have much to do with how much space the filesystem-as-file takes up on the underlying filesystem.

If you can accept setting an arbitrary maximum size limit (which can be much greater than the actual size of the underlying filesystem) for the filesystem-as-file, you can create a sparse file and a filesystem on it right now:

/tmp# df -h .
Filesystem                Size  Used Avail Use% Mounted on
<current filesystem>       20G   16G  3.0G  84% /

/tmp# dd if=/dev/null bs=1 seek=1024000000000 of=testdummy
0+0 records in
0+0 records out
0 bytes copied, 0.000159622 s, 0.0 kB/s
/tmp# ll testdummy
-rw-r--r-- 1 root root 1024000000000 Feb 19 08:24 testdummy
/tmp# ll -h testdummy
-rw-r--r-- 1 root root 954G Feb 19 08:24 testdummy

Here, I've created a file that appears to be a whole lot bigger than the filesystem it's stored onto...

/tmp# du -k testdummy
0       testdummy

...but so far it does not actually take any disk space at all (except for the inode and maybe some other metadata).

It would be perfectly possible to losetup it, create a filesystem on it and start using it. Each write operation that actually writes data to the file would cause the file's space requirement to grow. In other words, while the file size as reported by ls -l would stay that arbitrary huge number all the time, the actual space taken by the file on the disk as reported by du would grow.

And if you mount the filesystem-as-file with a discard mount option, shrinking can work automatically too:

/tmp# losetup /dev/loop0 testdummy
/tmp# mkfs.ext4 /dev/loop0
/tmp# mount -o discard /dev/loop0 /mnt
/tmp# du -k testdummy 
1063940 testdummy
/tmp# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      938G   77M  890G   1% /mnt

/tmp# cp /boot/initrd.img /mnt
/tmp# du -k testdummy 
1093732 testdummy

/tmp# rm /mnt/initrd.img
/tmp# du -k testdummy
1063944 testdummy

Automatic shrinking requires:

1.) that the filesystem type of the filesystem-as-file supports the discard mount option (so that the filesystem driver can tell the underlying system which blocks can be deallocated)

2.) and that the filesystem type of the underlying filesystem supports "hole punching", i.e. the fallocate(2) system call with the FALLOC_FL_PUNCH_HOLE option (so that the underlying filesystem can be told to mark some of the previously-allocated blocks of the filesystem-as-file as sparse blocks again)

3.) and that you're using kernel version 3.2 or above, so that the loop device support has the necessary infrastructure for this.

https://outflux.net/blog/archives/2012/02/15/discard-hole-punching-and-trim/

If you're fine with less immediate shrinking, you could just periodically run fstrim on the filesystem-as-file instead of using the discard mount option. If the underlying filesystem is very busy, avoiding immediate shrinking might help minimizing the fragmentation of the underlying filesystem.

The problem with this approach is that if the underlying filesystem becomes full, it won't be handled very gracefully. If there is no longer space in the underlying filesystem, the filesystem-as-file will start receiving errors when it's trying to replace sparse "holes" with actual data, even as the filesystem-as-file would appear to have some unused capacity left.

  • Very interesting! Other than fragmentation, how bad are the performance caveats when using discard or fstrim ? – Jay Feb 19 at 8:18
  • I haven't done comparative testing, but it seems generally reasonably usable if you don't need top performance. Of course it's going to depend rather heavily on filesystem choices, both on the filesystem-as-file and on the underlying filesystem, on the amount of other activity on the underlying filesystem, and what kind of hardware you're doing this on. – telcoM Feb 19 at 9:01
1

No, it is not supported automatically

Filesystems are created to handle a fixed amount of disk space. Some filesystems support growing, and fewer support shrinking, but not on demand without user intervention. An answer to the question you mention says that you have to do the resizing manually.

Of course what can be done manually also can be done automatically, but it would need support form the kernel that in case the file system is full it would not report the filesystem as full but enlarge the filesystem. You further want to shrink the filesystem automatically when you delete a file, this is potentially a very time consuming operation as you might have to move data around so that space at the end of the filesystem is free.

A filesystem created specifically for this task could just allocate blocks at the end on grow and free blocks in the middle on shrink, but there doesn't seem to be someone who wants to create such a file system. Of course you are free to do it yourself.

VirtualBox's Dynamic Disks are not the same. They are created with a fixed maximum size, they just don't immediately use up that amount of disk space. A block is allocated as soon as data is written to that block, and it doesn't shrink because that block is unused.

0

Using a sparse file + loop device + TRIM as in the accepted is nice and fine, however I'm not really convinced yours is not an XY problem: why not simply use a subdirectory/subtree? That will grow and shrink as needed, and you could also mount in another place either with mount --bind or NFS ;-)

I am also aware of existing systems that do something similar: specifically VirtualBox's Dynamic Disks. If there is some way of using those - or something akin to them - without actually running a virtual machine, I would also be happy with that.

With qemu-ndb, you can also use in the host system any disk image supported by qemu:

# modprobe nbd
# qemu-nbd --discard=unmap -c /dev/nbd1 image.qcow2
# kpartx -a /dev/nbd1
# mount /dev/mapper/nbd1p1 /mount/point
...
# fstrim -v /mount/point
...
# qemu-nbd -d /dev/nbd1

Qemu's qcow2 images not only grow and shrink to accomodate the data, they also support snapshots & "diff" images.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.