I'm wondering how other people feel about forward symlinks: are they safe? Good practice? Depends? [Edit] -- I didn't define forward symlinks as I was too close to the problem space to see that term might not be well understood.

With the mostly failed attempt to merge root and /usr (I don't know of any distro (except cygwin if you count that as a distro) that has gotten rid of two separate directories for bin, lib and lib64), the distro I mostly use is OpenSUSE. Their practice moving forward to implement the merge has been to install the binaries in /usr. Since so many progs have hard coded paths for many programs (/bin/ comes to mind as probably the most common), the install package puts a symlink from /bin/prog -> /usr/bin/prog. That alone is not a forward link (though the term could be defined to mean from an upper level directory to a director at a lower level tree, but that's obviously a non-issue in many environments.

In environments concerned w/security and reliability, different trees are placed on different devices to 1) optimize backup & restore time and backup space. 2) to contain problems that wipe or corrupt a partition & 3) to disallow some security attacks that might allow hardlinked files, among other mechanisms, to be used in an attack. Related to reason 1, some directories and descendents get many frequent small changes (/var). Some are changed infrequently (/etc, /bin, /sbin). Some may get daily changes (/home). Grouping similarly used directories on 1 partition may allow less frequent backup of infrequently changed partitions, while frequently changed directories might be candidates for placing on a ram-disk or SSD.

However, reasons for different partitions are not relevant to the practice of forward linking being safe or a "best practice" (or not). To be more explicit -- in the case of putting symlinks in /bin, /sbin and /lib64 pointing to directories under /usr/{{s,}bin,lib64}, if "/usr" is a separate partition, then I assert the links in /{{s,}bin,lib64/ are "unsafe" and actually poor administrative practice, since no matter how reliable your disks are that hold /usr, if for some reason they cannot be mounted -- then your binary and library links in the root dirs /{bin,sbin,lib64} are worthless and will prevent the system from booting.

I've personally experienced three cases - the binary for 'mount' being on /usr/bin/mount with a symlink in /bin/mount, with newer mount progs that have been restructured into several libraries, and symlinks for libraries in /lib64 pointed to a yet-to-be-mounted /usr/lib64, and the most recent failure mode: placing library version numbers inside the binaries and libraries that prevent a program loading w/the wrong numbered lib version.

More often than not, such a paring will work. The main problem is you don't know they've embedded version numbers until the paired directory isn't mounted. During normal operations, files on the root can be dynamically linked with files in /usr/lib64. If /usr doesn't mount, it may try to link with the same named-library located in /lib64 -- and then fail. That last case is a side effect of trying to undo symlinks in the root partition to /usr by copying the same-named libs in /usr/lib64 to /lib64, which. Unfortunately when updating with a new version of the same package, the new version doesn't get copied to the root dir. If the time/date stamp on the on the /usr/lib64 copy is older than the one in /lib64, there's no trivial way to catch that.

The linker util 'ldd' shows required libs that have filename based-versions but doesn't immediately know about the embedded versions. All this extra work caused by placing needed resources needed to mount and boot other file systems on the file systems yet to be mounted.

I strongly think that a reason no one has pointed this out as bad practice is either no one believing that someone would implement something so failure prone, or simply believing the newly created reason that it isn't a "bug" -- that separate partitions (as originally setup by earlier versions of the SuSE installer) are no longer supported, just as fast booting directly from the hard disk is no longer supported (even though recommended by systemd devs to speed up boot times).

Unfortunately, the new desktop devs disregard previous administrative practice and at the same time don't do the safer merge of /usr/bin files to /bin, with some giving the reason that there might not be enough room on root!

I'm unable to get any reasons for not merging them like cygwin has done (they simply mount /bin @ /usr/bin -- no symlinks involved). Instead I'm told to stop arguing and just accept it, which seems much like the somewhat sadistic advice to to "just accept" other problematic behaviors/events.

Previous end sentence: "Am I being overly conservative or are forward link now considered to be in the realm of 'good practice' for critical files??"

Given the new ways to fail that arise, I no longer consider the possibility of me being overly conservative in regards to this issue. I can't see a way it can be justified as administrative "best practice".

migrated from serverfault.com Apr 18 '13 at 14:47

This question came from our site for system and network administrators.

  • I have never run into a system with /usr on its own partition... since it is for userland tools which are necessary to get to the system booted (just like /etc contains config files), I don't see why you would do this. In which case, symlinking would be find. Is it common to slice /usr into its own partition? – Matt Apr 17 '13 at 23:13
  • 1
    I certainly mount /usr as a separate partition occasionally. – fukawi2 Apr 17 '13 at 23:38
  • 1
    Mounting "/usr" separately, isn't as common in the desktop world, but used to be standard fair in the serverworld (where this was first posted). Servers need higher reliability and often don't boot to a GUI or 'X'. – Astara Apr 19 '13 at 19:47
  • 1
    @Matt: So you've never used OpenBSD? – Martin Schröder Apr 23 '13 at 20:55

The argument is that figuring out where a given binary should be ( / or /usr ) is needless complication that serves no purpose. Systems have already not been able to boot without /usr for some time now so there is no longer a reason to keep two directories.

See http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge for more details.

  • "Systems have not been able to boot without /usr for some time now"? – Astara Apr 19 '13 at 21:10
  • You might want to qualify that. What do you mean by "boot" and what do you mean by "Systems". Certainly some systems is a more appropriate statement that just "systems". But that also depends on what you mean by 'boot'. More specifically -- Boot is anything it takes to get the machine to run-level 1. -- single user with networking no services other than, perhaps sshd. So what is needed on /usr for that to happen? Now If you want to call "starting services" and starging a graphical user interface" "boot"... You might as well toss in starting up facebook, etc. – Astara Apr 19 '13 at 21:17
  • BTW -- having glanced at that webpage (last time I looked it was down).. I see alot of myths called fact and facts called myths. One can make any argument one wants if one allows calling "black", "white", and vice-versa. Wasn't it Orwell that called that "newspeak".. – Astara Apr 19 '13 at 21:19
  • @Astara, udev has relied on numerous components in /usr for some time, so even single user mode can not function without it. systemd requires /usr as well. – psusi Apr 19 '13 at 21:56
  • 1
    This is a rather simplistic view typical of Linux (or should I say Ubuntu?); the OpenBSD way is to have more partitions, not fewer. – Martin Schröder Apr 23 '13 at 20:55

A better reason for extensive (careful) use of symlinks...

Symbolic links do for filesystems (a kind of hierarchical database) what foreign keys do for relational databases.

With judicious use of both "forward" and "backward" symlinks (though I've not heard them referred to that way before), a distro could be designed where most things in /etc, /bin, /lib, and /sbin are symlinks to /usr/etc, /usr/bin, /usr/lib, and /usr/sbin. There could then be multiple versions of the /usr directory mounted someplace like under /initrd (initialize ramdisk). The filesystem could then manage, through careful creation and deletion of symlinks, which version was being used for each file at any given time.

Puppy Linux and some other distros use the unionfs and aufs filesystems to implement a variation of this concept.

The original O/S, as distributed, is kept static (unaltered) in permanent storage as a "bottom layer". When one of the files, such as /etc/hosts for example, is edited and saved, instead of changing the original, the filesystem creates a new copy in the topmost "working layer" on ramdisk. The filesystem then presents this copy, instead of the original, to the user.

The ramdisk copies are periodically (root user configurable) flushed in the background to a third layer which is also on permanent storage. Altered, ramdisk copies effectively overlay their saved versions which in turn overlay the static originals. Only the topmost copy is visible (for an unaltered file, this would be the original) so the file system looks completely typical to the user and other software.

This technique improves system speed and reliability...

  • User-initiated file reads and writes are very fast since they always use ramdisk.
  • Since the ramdisk layer only contains files recently altered, it stays small and functions like another cache.
  • Flush to slower permanent storage is deferred to background processing.
  • Copies can be made periodically of the "savefile" third layer, thus providing an "undo" capability for when configs or installs go wrong or malicious programs are detected.

Symlinks are what make all of this possible.

Answers to Questions...

How "high speed" is your setup?

The more memory you can give it, the more program can be kept in the ramdisk and thus the more responsive it will be. Starting a program from ramdisk is a bit faster than starting it from flash-memory (flashdrive, SD, etc.) and only takes a tiny fraction of the time needed to start the same program from a hard drive.

On my 300MHz 1999 Toshiba 4030CDT laptop with 64MB RAM, Puppy Linux 5.2.2 Wary, mostly based on Slackware, there's no room for much ramdisk so programs load from hard drive. Still, the 2D-GUI is quite responsive. I use it as the "console" connected via Synergy to all the other hosts.

At the other end of the scale is what I'm using now (via the laptop)... a Compaq S6010V running a 2.6GHz Celeron processor and 1.3GB RAM. The ramdisk "PuppySpace" has been allotted 512MB of which less than 200MB is currently in use. Loaded are Zim (a Python note-taking app), Geany editor/IDE, a terminal client with 5 sessions open, and 2 copies of Chromium with a total of 12 tabs of web pages active including Gmail.

how long does it normally stay up?

Because I'm constantly developing and changing configs, planned reboots are common. The uptime output for the Compaq is currently...

16:21:10 up 4 days,  7:28, load average: 0.06, 0.24, 0.30

Is it easy for the user to compile their own kernel from kernel.org and boot with it?

I can't address that as I've never done it. Though the Puppy Linux community is full of people that compile their own kernels for breakfast.

Is it something you run servers from?

All my systems are running either JWM or Openbox window managers (GTK+ based) but some work has been done setting up Puppy as a server as with LEMP and the Simplified Music Server Jukebox (mpdPup).

aufs? ...what's it good for? (over xfs/ext.

Aufs is a complete rewrite of unionfs. They both implement a union mount where multiple filesystems like xfs, ext3/4, etc. are mounted to the same mountpoint so they overlay each other.

  • I'd say correction: using symlinks is one possible way of implementing something like that. – a CVn Apr 19 '13 at 19:28
  • I'd say you have a different problem space. You are running off of a root ram-device where everything is rooted in ram. How "high speed" is your setup" .. how long does it normally stay up? Is it easy for the user to compile their own kernel from kernel.org and boot with it? Is it something you run servers from? This was posted in the server forum initially, because the sytem I was talking about is a server that doesn't start a GUI -- just services. I've wanted to test and play w/unionfs -- aufs? Don't know anything about that one though...what's it good for? (over xfs/ext<N>... et – Astara Apr 19 '13 at 21:30
  • I am not sure how those implementations worked, but their successor (overlayfs) doesn't use file-system symbolic links. Second point - I find it hard to believe they used file-system symlinks, as many programs will detect them as possibly being part of an insecure path. Third point -- in the implementations I've seen, the top-fs (the new stuff) sat on top of 1 lower fs. Otherwise too many things didn't work right -- like hard links that wouldn't appear to be on separate devices but actually are. As a result, there were no forward/backward symlinks at least not how I am using the terms. – Astara Jun 24 at 19:24
  • I think terminology is tripping us up some here. The topmost directory / is called "the root directory" but there is also a /root directory under that in most distros that is the analog of the "home directory" (i.e. /home/username) for the user named root. There should never be files in the / directory, only subdirectories. – DocSalvager Jun 25 at 2:02
  • The terms "forward" and "backward" in relation to symlinks are confusing. A valid (e.g. "unbroken") symlink always points to an existing file, directory or another link. In that sense, it could be said to be "a backward link". But in practice, most people think of it as a "forward link." Regarding safety... remember that symlinks have absolutely nothing to do with security. Any user can create a symlink pointing to anything. But the link will be "broken" if the user doesn't have permission to read the target or the target doesn't exist. Things on an unmounted partition don't exist yet. – DocSalvager Jun 25 at 2:15

Right now it's best to not make /usr a separate partition anymore. This way there is no need for symlinks at all.

At least, that's what I decided after the problems with binaries moving around started in my distribution. It actually makes a lot of sense to me, to have all binaries and libraries on the same partition. It doesn't matter then which are in /bin and which are in /usr/bin, and they can be moved around as you please.

The only reason I had a separate /usr in the first place was that I simply made a partition for everything; with LVM, it's easy to do. However apart from that, there was no practical reason for it. It was separated for the sake of separating.

If you have a partition for everything, your root / partition will be pretty much entirely empty. In my case it was less than 200MB. So I merged it with the /usr partition and freed the space previously uselessly wasted by the root partition. So far I have not discovered any downsides.

The merging process from a rescue system: (use at your own risk - may break your system)

mkdir /mnt/root /mnt/usr
mount /dev/lvm/root /mnt/root
mount /dev/lvm/usr /mnt/usr
# /usr will be mounted as /, so move everything into a new usr/ subdirectory
mkdir /mnt/usr/usr
mv /mnt/usr/* /mnt/usr/usr/
# copy the root files, preserving hard links just in case
rsync -aH /mnt/root/. /mnt/usr/.
# update fstab: comment /usr, change UUID of / to the one of /usr
nano /mnt/usr/etc/fstab
umount /mnt/root
umount /mnt/usr
lvrename lvm/root lvm/oldroot
lvrename lvm/usr lvm/root
# chroot to see if it works and update grub UUID
mount /dev/lvm/root /mnt/root
mount -o bind /dev /mnt/root/dev
mount -o bind /proc /mnt/root/proc
mount -o bind /sys /mnt/root/sys
chroot /mnt/root /bin/bash
update-grub # may depend on distro/bootloader
  • Ok -- question about the above -- the systemd page recommends getting rid of initrd in order to optimize boot time. I have been running w/o a initrd for -- "forever"... (I tried it, and it made debugging boot problems near impossible)...so does the above work with without initrd -- as having to load a bunch of utils from my hard disk onto a ramdisk in order to boot the system where where those utils worked in the first place seems a bit silly for a single user (for a distro that can't predict HW, that makes sense but...)... – Astara Apr 19 '13 at 21:24
  • If your /usr is on LVM, the above would result in a root on LVM setup. Which is usually booted using Initramfs. As for boot time, there should be no big difference. It takes time to scan and enable LVM volumes; but whether that happens at Initramfs stage or real root stage, doesn't matter. If your Initramfs significantly slows boot time, something's wrong. – frostschutz Apr 19 '13 at 22:29
  • Idea of merging / & /usr into 1 partition isn't what my question was about it was about safety of having pointers in /bin+/lib64 pointing to binaries on separate partition "/usr" w/same named directories. So /bin/mount would point to /usr/bin/mount, where /usr isn't mounted yet -- that is a forward link. You need some partition to boot from. Using a ramfs puts extra copies of this utils on a ramdisk -- so you will read in the common utils twice. Once for ram disk and another on real disk when it is accessed. Anything on ramdisk will be slower to read in as its done via BIOS calls. – Astara Jun 24 at 19:49

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.