forward vs. backward symlinks: administrative practice?

Question

I'm wondering how other people feel about forward symlinks: are they safe? Good practice? Depends? [Edit] -- I didn't define forward symlinks as I was too close to the problem space to see that term might not be well understood.

With the mostly failed attempt to merge root and /usr (I don't know of any distro (except cygwin if you count that as a distro) that has gotten rid of two separate directories for bin, lib and lib64), the distro I mostly use is OpenSUSE. Their practice moving forward to implement the merge has been to install the binaries in /usr. Since so many progs have hard coded paths for many programs (/bin/ comes to mind as probably the most common), the install package puts a symlink from /bin/prog -> /usr/bin/prog. That alone is not a forward link (though the term could be defined to mean from an upper level directory to a director at a lower level tree, but that's obviously a non-issue in many environments.

In environments concerned w/security and reliability, different trees are placed on different devices to 1) optimize backup & restore time and backup space. 2) to contain problems that wipe or corrupt a partition & 3) to disallow some security attacks that might allow hardlinked files, among other mechanisms, to be used in an attack. Related to reason 1, some directories and descendents get many frequent small changes (/var). Some are changed infrequently (/etc, /bin, /sbin). Some may get daily changes (/home). Grouping similarly used directories on 1 partition may allow less frequent backup of infrequently changed partitions, while frequently changed directories might be candidates for placing on a ram-disk or SSD.

However, reasons for different partitions are not relevant to the practice of forward linking being safe or a "best practice" (or not). To be more explicit -- in the case of putting symlinks in /bin, /sbin and /lib64 pointing to directories under /usr/{{s,}bin,lib64}, if "/usr" is a separate partition, then I assert the links in /{{s,}bin,lib64/ are "unsafe" and actually poor administrative practice, since no matter how reliable your disks are that hold /usr, if for some reason they cannot be mounted -- then your binary and library links in the root dirs /{bin,sbin,lib64} are worthless and will prevent the system from booting.

I've personally experienced three cases - the binary for 'mount' being on /usr/bin/mount with a symlink in /bin/mount, with newer mount progs that have been restructured into several libraries, and symlinks for libraries in /lib64 pointed to a yet-to-be-mounted /usr/lib64, and the most recent failure mode: placing library version numbers inside the binaries and libraries that prevent a program loading w/the wrong numbered lib version.

More often than not, such a paring will work. The main problem is you don't know they've embedded version numbers until the paired directory isn't mounted. During normal operations, files on the root can be dynamically linked with files in /usr/lib64. If /usr doesn't mount, it may try to link with the same named-library located in /lib64 -- and then fail. That last case is a side effect of trying to undo symlinks in the root partition to /usr by copying the same-named libs in /usr/lib64 to /lib64, which. Unfortunately when updating with a new version of the same package, the new version doesn't get copied to the root dir. If the time/date stamp on the on the /usr/lib64 copy is older than the one in /lib64, there's no trivial way to catch that.

The linker util 'ldd' shows required libs that have filename based-versions but doesn't immediately know about the embedded versions. All this extra work caused by placing needed resources needed to mount and boot other file systems on the file systems yet to be mounted.

I strongly think that a reason no one has pointed this out as bad practice is either no one believing that someone would implement something so failure prone, or simply believing the newly created reason that it isn't a "bug" -- that separate partitions (as originally setup by earlier versions of the SuSE installer) are no longer supported, just as fast booting directly from the hard disk is no longer supported (even though recommended by systemd devs to speed up boot times).

Unfortunately, the new desktop devs disregard previous administrative practice and at the same time don't do the safer merge of /usr/bin files to /bin, with some giving the reason that there might not be enough room on root!

I'm unable to get any reasons for not merging them like cygwin has done (they simply mount /bin @ /usr/bin -- no symlinks involved). Instead I'm told to stop arguing and just accept it, which seems much like the somewhat sadistic advice to to "just accept" other problematic behaviors/events.

Previous end sentence: "Am I being overly conservative or are forward link now considered to be in the realm of 'good practice' for critical files??"

Given the new ways to fail that arise, I no longer consider the possibility of me being overly conservative in regards to this issue. I can't see a way it can be justified as administrative "best practice".

Answer 1

The argument is that figuring out where a given binary should be ( / or /usr ) is needless complication that serves no purpose. Systems have already not been able to boot without /usr for some time now so there is no longer a reason to keep two directories.

See http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge for more details.

Answer 2

A better reason for extensive (careful) use of symlinks...

Symbolic links do for filesystems (a kind of hierarchical database) what foreign keys do for relational databases.

With judicious use of both "forward" and "backward" symlinks (though I've not heard them referred to that way before), a distro could be designed where most things in /etc, /bin, /lib, and /sbin are symlinks to /usr/etc, /usr/bin, /usr/lib, and /usr/sbin. There could then be multiple versions of the /usr directory mounted someplace like under /initrd (initialize ramdisk). The filesystem could then manage, through careful creation and deletion of symlinks, which version was being used for each file at any given time.

Puppy Linux and some other distros use the unionfs and aufs filesystems to implement a variation of this concept.

The original O/S, as distributed, is kept static (unaltered) in permanent storage as a "bottom layer". When one of the files, such as /etc/hosts for example, is edited and saved, instead of changing the original, the filesystem creates a new copy in the topmost "working layer" on ramdisk. The filesystem then presents this copy, instead of the original, to the user.

The ramdisk copies are periodically (root user configurable) flushed in the background to a third layer which is also on permanent storage. Altered, ramdisk copies effectively overlay their saved versions which in turn overlay the static originals. Only the topmost copy is visible (for an unaltered file, this would be the original) so the file system looks completely typical to the user and other software.

This technique improves system speed and reliability...

• User-initiated file reads and writes are very fast since they always use ramdisk.
• Since the ramdisk layer only contains files recently altered, it stays small and functions like another cache.
• Flush to slower permanent storage is deferred to background processing.
• Copies can be made periodically of the "savefile" third layer, thus providing an "undo" capability for when configs or installs go wrong or malicious programs are detected.

Symlinks are what make all of this possible.

Answers to Questions...

How "high speed" is your setup?

The more memory you can give it, the more program can be kept in the ramdisk and thus the more responsive it will be. Starting a program from ramdisk is a bit faster than starting it from flash-memory (flashdrive, SD, etc.) and only takes a tiny fraction of the time needed to start the same program from a hard drive.

On my 300MHz 1999 Toshiba 4030CDT laptop with 64MB RAM, Puppy Linux 5.2.2 Wary, mostly based on Slackware, there's no room for much ramdisk so programs load from hard drive. Still, the 2D-GUI is quite responsive. I use it as the "console" connected via Synergy to all the other hosts.

At the other end of the scale is what I'm using now (via the laptop)... a Compaq S6010V running a 2.6GHz Celeron processor and 1.3GB RAM. The ramdisk "PuppySpace" has been allotted 512MB of which less than 200MB is currently in use. Loaded are Zim (a Python note-taking app), Geany editor/IDE, a terminal client with 5 sessions open, and 2 copies of Chromium with a total of 12 tabs of web pages active including Gmail.

how long does it normally stay up?

Because I'm constantly developing and changing configs, planned reboots are common. The uptime output for the Compaq is currently...

16:21:10 up 4 days,  7:28, load average: 0.06, 0.24, 0.30

Is it easy for the user to compile their own kernel from kernel.org and boot with it?

I can't address that as I've never done it. Though the Puppy Linux community is full of people that compile their own kernels for breakfast.

Is it something you run servers from?

All my systems are running either JWM or Openbox window managers (GTK+ based) but some work has been done setting up Puppy as a server as with LEMP and the Simplified Music Server Jukebox (mpdPup).

aufs? ...what's it good for? (over xfs/ext.

Aufs is a complete rewrite of unionfs. They both implement a union mount where multiple filesystems like xfs, ext3/4, etc. are mounted to the same mountpoint so they overlay each other.

Answer 3

Right now it's best to not make /usr a separate partition anymore. This way there is no need for symlinks at all.

At least, that's what I decided after the problems with binaries moving around started in my distribution. It actually makes a lot of sense to me, to have all binaries and libraries on the same partition. It doesn't matter then which are in /bin and which are in /usr/bin, and they can be moved around as you please.

The only reason I had a separate /usr in the first place was that I simply made a partition for everything; with LVM, it's easy to do. However apart from that, there was no practical reason for it. It was separated for the sake of separating.

If you have a partition for everything, your root / partition will be pretty much entirely empty. In my case it was less than 200MB. So I merged it with the /usr partition and freed the space previously uselessly wasted by the root partition. So far I have not discovered any downsides.

The merging process from a rescue system: (use at your own risk - may break your system)

mkdir /mnt/root /mnt/usr
mount /dev/lvm/root /mnt/root
mount /dev/lvm/usr /mnt/usr
# /usr will be mounted as /, so move everything into a new usr/ subdirectory
mkdir /mnt/usr/usr
mv /mnt/usr/* /mnt/usr/usr/
# copy the root files, preserving hard links just in case
rsync -aH /mnt/root/. /mnt/usr/.
# update fstab: comment /usr, change UUID of / to the one of /usr
nano /mnt/usr/etc/fstab
umount /mnt/root
umount /mnt/usr
lvrename lvm/root lvm/oldroot
lvrename lvm/usr lvm/root
# chroot to see if it works and update grub UUID
mount /dev/lvm/root /mnt/root
mount -o bind /dev /mnt/root/dev
mount -o bind /proc /mnt/root/proc
mount -o bind /sys /mnt/root/sys
chroot /mnt/root /bin/bash
update-grub # may depend on distro/bootloader