9

Problem: NFS can be slow, when starting binaries (e.g. /usr/bin) over NFS, such as in a network booted system. RAM buffer cache might not be sufficient to avoid slowness.

Idea: it seems we should be able to have a local disk cache which would save the file(s) locally as they are pulled from NFS.

Question: has anyone seen anything like that on any UNIX system?

Background:

In FreeBSD there are great ways of using a unionfs to build amazing stacked file systems. I presently have a system on AWS which uses only 1 GB of disk because it has the majority of the /usr file system tree mounted through NFS. In the old days you could do that easily because /usr was not required for a basic boot. Now it's much harder (especially on AWS where you can't bail out to a console when booting fails) but I managed by having the minimum necessary stuff from the /usr tree on the local drive, and then, when network is started, I mount NFS over the /usr tree.

I even have a back-door where I can still write to the underlying minimal local hard drive /usr tree in case I need to update something on the running system.

It's beautiful.

Except that the NFS (Amazon EFS) is very slow. And buffer caching is not working well enough. The aws command line interface to manage AWS resources, for example, uses Python, and there is a sh*tton of includes sucked in every time the aws command is called. It takes 20 seconds to run a simple aws CLI command. And even running it repeatedly, you'd think that caching might help, NFS attribute caching, etc, but it doesn't.

Possible Solution (on FreeBSD):

So what I'd like to do is put another unionfs layer on top of the NFS layer, which is a local disk based UFS file system. But it would start out empty on boot, and then, every time we load anything from NFS (assuming now that it's stable binaries, not dynamic updated data) it would leave a copy on the disk.

Implementation of this Solution:

So here is what I'm thinking should be done. In /usr/src/sys/fs/unionfs/union_vnops.c we have this very simple code:

static int
unionfs_open(struct vop_open_args *ap)
{
    ...
    if (targetvp == NULLVP) {
        if (uvp == NULLVP) {
            if ((ap->a_mode & FWRITE) && lvp->v_type == VREG) {
                error = unionfs_copyfile(unp,
                    !(ap->a_mode & O_TRUNC), cred, td);
                if (error != 0)
                    goto unionfs_open_abort;
                targetvp = uvp = unp->un_uppervp;
            } else
                targetvp = lvp;
        } else
            targetvp = uvp;
    }

This is the part which will make a copy on the upper layer if we are accessing a file for writing (ap->a_mode & FWRITE) that is only on the lower layer (uvp == NULLVP) && lvp->v_type == VREG.

It seems simple enough to want to try to add a feature which would create a copy for every file, even one accessed for read only. It would then also make that copy, and next time we would read that file from the disk.

To do that I would add a new option in /usr/src/sys/fs/unionfs/union.h I would add a new option, the copy policy:

/* copy policy of upper layer */
typedef enum _unionfs_copypolicy {
       UNIONFS_COPY_ON_WRITE = 0,
       UNIONFS_COPY_ALWAYS
} unionfs_copypolicy;

struct unionfs_mount {
    struct vnode   *um_lowervp; /* VREFed once */
    struct vnode   *um_uppervp; /* VREFed once */
    struct vnode   *um_rootvp;  /* ROOT vnode */
    unionfs_copypolicy um_copypolicy;
    unionfs_copymode um_copymode;
    unionfs_whitemode um_whitemode;
    uid_t       um_uid;
    gid_t       um_gid;
    u_short     um_udir;
    u_short     um_ufile;
};

and frankly, I'd like to handle all these modes as a bit field for space. Anyway, with this, I can now change the above code to:

unp = VTOUNIONFS(ap->a_vp);
ump = MOUNTTOUNIONFSMOUNT(ap->a_vp->v_mount);
...

    if (targetvp == NULLVP) {
        if (uvp == NULLVP) {
            if (((ap->a_mode & FWRITE) || (ump->um_copypolicy == UNIONFS_COPY_ALWAYS)) && lvp->v_type == VREG) {
                error = unionfs_copyfile(unp,
                    !(ap->a_mode & O_TRUNC), cred, td);
                if (error != 0)
                    goto unionfs_open_abort;
                targetvp = uvp = unp->un_uppervp;

This should be all that's needed. That is, hoping that all that dealing with attributes and shadow directories is handled from inside the function unionfs_copyfile, as it should.

In that case now, we only would need to add the new copy-on-read policy option into mount_unionfs which is nicely inside the kernel module also /usr/src/sys/fs/unionfs/union_vfsops.c

static int
unionfs_domount(struct mount *mp)
{
    int     error;
    ...
    u_short     ufile;
    unionfs_copypolicy copypolicy;
    unionfs_copymode copymode;
    unionfs_whitemode whitemode;
    ...
    ufile = 0;
    copypolicy = UNIONFS_COPY_ON_WRITE; /* default */
    copymode = UNIONFS_TRANSPARENT; /* default */
    whitemode = UNIONFS_WHITE_ALWAYS;
    ...
        if (vfs_getopt(mp->mnt_optnew, "copypolicy", (void **)&tmp,
            NULL) == 0) {
            if (tmp == NULL) {
                vfs_mount_error(mp, "Invalid copy policy");
                return (EINVAL);
            } else if (strcasecmp(tmp, "always") == 0)
                copypolicy = UNIONFS_COPY_ALWAYS;
            else if (strcasecmp(tmp, "onwrite") == 0)
                copypolicy = UNIONFS_COPY_ON_WRITE;
            else {
                vfs_mount_error(mp, "Invalid copy policy");
                return (EINVAL);
            }
        }

        if (vfs_getopt(mp->mnt_optnew, "copymode", (void **)&tmp,
            ...
        }
        if (vfs_getopt(mp->mnt_optnew, "whiteout", (void **)&tmp,
            ...
        }
    }
    ...

    UNIONFSDEBUG("unionfs_mount: uid=%d, gid=%d\n", uid, gid);
    UNIONFSDEBUG("unionfs_mount: udir=0%03o, ufile=0%03o\n", udir, ufile);
    UNIONFSDEBUG("unionfs_mount: copypolicy=%d, copymode=%d, whitemode=%d\n", copypolicy, copymode, whitemode);

So, this would do what I want in FreeBSD, I need to now get the sources for my system, apply this patch, recompile the unionfs.ko kernel module and swap it into my system and see if it will work.

# Custom /etc/fstab for FreeBSD VM images
/dev/gpt/rootfs  /        ufs      rw      1       1
/dev/gpt/varfs   /var     ufs      rw      1       1
fdesc            /dev/fd  fdescfs  rw      0       0
proc             /proc    procfs   rw      0       0
/usr             /.usr    nullfs   rw      0       0
fs-xxxxxxxx.efs.rrrr.amazonaws.com:/ /usr nfs rw,nfsv4,minorversion=1,oneopenown,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,late,bg 0 0
/var/cache/usr   /usr     unionfs rw,copypolicy=always 0 0

More Improvements: Evict Cache Entries

Now I notice that I might want to add another whiteout mode, which is: never. I.e., I should be able to delete a file from the upper layer with the effect of evicting the file from the cache but no whiteout effect of masking the file from the lower layer so it appears empty. This is how it could be done in union.h add the UNIONFS_WHITE_NEVER:

/* whiteout policy of upper layer */
typedef enum _unionfs_whitemode {
       UNIONFS_WHITE_ALWAYS = 0,
       UNIONFS_WHITE_WHENNEEDED,
       UNIONFS_WHITE_NEVER
} unionfs_whitemode;

and then in union_vnops.c:

static int
unionfs_remove(struct vop_remove_args *ap)
{
    ...
    if (uvp != NULLVP) {
        /*
         * XXX: if the vnode type is VSOCK, it will create whiteout
         *      after remove.
         */
        if (ump == NULL || ump->um_whitemode == UNIONFS_WHITE_ALWAYS ||
            (lvp != NULLVP && ump->um_whitemode != UNIONFS_WHITE_NEVER))
            cnp->cn_flags |= DOWHITEOUT;
        error = VOP_REMOVE(udvp, uvp, cnp);
    } else if (lvp != NULLVP && ump->um_whitemode != UNIONFS_WHITE_NEVER)
        error = unionfs_mkwhiteout(udvp, cnp, td, path);

Then there might be something about rmdir also.

static int
unionfs_rmdir(struct vop_rmdir_args *ap)
{
    ...
    if (uvp != NULLVP) {
        if (lvp != NULLVP) {
            error = unionfs_check_rmdir(ap->a_vp, cnp->cn_cred, td);
            if (error != 0)
                return (error);
        }
        ump = MOUNTTOUNIONFSMOUNT(ap->a_vp->v_mount);
        if (ump->um_whitemode == UNIONFS_WHITE_ALWAYS || 
            (lvp != NULLVP && ump->um_whitemode != UNIONFS_WHITE_NEVER))
            cnp->cn_flags |= DOWHITEOUT;
        error = unionfs_relookup_for_delete(ap->a_dvp, cnp, td);
        if (!error)
            error = VOP_RMDIR(udvp, uvp, cnp);
    }
    else if (lvp != NULLVP && ump->um_whitemode != UNIONFS_WHITE_NEVER)
        error = unionfs_mkwhiteout(udvp, cnp, td, unp->un_path);

This should do the eviction stuff too.

But before I do this all, I wonder, is there an existing trick that people have already figured out instead?


PS: here is my full diff and test results. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251363

The short answer is: it actually works nicely, and another thing I wasn't clear about: the unionfs doesn't take a block device but it takes a directory! So that's really cool that you don't even need to create a device. I have updated my proposed fstab, although I would probably not use that at all, since it must be delayed until after the late mount of the NFS. So better to remove that and turn this unionfs based cache on later, such as in /etc/rc.local, it's so easy:

mount -t unionfs -o copypolicy=always /var/cache/usr /usr

I also found that the /var/cache/usr directory remains directly available, so the eviction from the cache can be done simply by removing files from there! This means we don't even need to mess with the whiteout settings at all.

Instead I should come up with an automatic cache eviction strategy to remove files by old atime from the cache if the unionfs_copyfile(...) call returns error "no space left on device", evict old files until space reclaimed, then retry the operation. Pretty easy (except for finding the files with old atime).

Poor Man's Easy Cache Eviction

Just run find /var/cache/usr -atime 2 -exec rm \{\}\; every few days to remove those items which haven't been accessed for a day.

A more interesting deeper issue might be if one could make the unionfs_copyfile(...) function more efficient, by writing the blocks to the upper layer as they are being read. Maybe even do the entire thing block-oriented, so that if the file in the lower layer is sparse, it would be kept sparse on the upper layer too.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.