1

I am using a virtual machine on a server I cannot access and I'm having the filesystem mounted in read-only mode:

# dmesg
.... 
[2.535658] EXT4-fs (vda): Couldn't remount RDWR because of unprocessed orphan inode list.  Please umount/remount instead

So I tried to run:

# e2fsck -f /dev/vda
e2fsck 1.42.9 (28-Dec-2013)
/dev/vda has unsupported feature(s): metadata_csum
e2fsck: Get a newer version of e2fsck!

Output of df:

/dev/root        41022688 37964956    944196  98% /
devtmpfs          4084008        0   4084008   0% /dev
tmpfs             4085752        0   4085752   0% /dev/shm
tmpfs             4085752   397012   3688740  10% /run
tmpfs             4085752        0   4085752   0% /sys/fs/cgroup
tmpfs              817152        0    817152   0% /run/user/0
tmpfs              817152        0    817152   0% /run/user/1000

Output of mount:

/dev/vda on / type ext4 (ro,relatime,stripe=8191,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=4084008k,nr_inodes=1021002,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=817152k,mode=700)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=817152k,mode=700,uid=1000,gid=1000)

Of course, I cannot update e2fsck, as I have a read-only filesystem. How can I escape the loop?

  • (1) I tried to improve the formatting of your terminal commands.  Please verify that I got it right (I had to guess where the linebreaks were supposed to be).  (2) You say “the filesystem”.  Does the VM operating system have only one filesystem?  It might help if you posted the output of df and mount.  … … … … … … … … … … … … … … … … … … … … … … … … … … …  Please do not respond in comments; edit your question to make it clearer and more complete. – G-Man May 29 '18 at 12:21
0

It looks like you have a /run ramdisk (tmpfs filesystem) that’s mounted read/write.  Can you download the more current e2fsck there?  You should then be able to

  • Run e2fsck -f /dev/vda from /run and fix your root filesystem.
  • Reboot.  This will wipe out your newly downloaded e2fsck, because tmpfs filesystems are re-initialized when you reboot.  But your root filesystem should be read/write now.
  • Download e2fsck again, onto the root filesystem.

In fact, if you can remount the root filesystem after you fix it (without rebooting), then you should just be able to copy e2fsck to /sbin (or wherever it belongs).

  • Since I am not a sysadmin (but an application software developer), I think I might need a bit more step-by-step support, if you could. In particular, I downloaded e2fsck and ran configure, but I have this error: "mkdir: cannot create directory '/tmp/cg22544-24946': Read-only file system mkdir: cannot create directory '/tmp/cg-22544': Read-only file system config.guess: cannot create a temporary directory in /tmp configure: error: cannot guess build type; you must specify one " – Manu May 29 '18 at 14:43
  • Can you download a precompiled copy of e2fsck? – G-Man May 29 '18 at 14:47
  • I'd like to get one, but where? – Manu May 29 '18 at 15:37
  • @Manu what distribution of Linux are you running (e.g. RHEL 7, Ubuntu 16.04)? – Mark Plotnick May 29 '18 at 17:47
  • CentosOs 7, but I did find it. Now I have if, but I run it, the error is ./e2fsck: symbol lookup error: ./e2fsck: undefined symbol: ext2fs_xattrs_read – Manu May 29 '18 at 19:44
0

The answer depends on how did this error come into being?

It might happen if you booted some newer kernel, and it upgraded your ext4 filesystem to newer version, which your default kernel does not fully support. If so, the solution may be as simple as boot again into that newer kernel, and cleanly unmount filesystem.

However if you "did nothing" and error appeared just by itself, it may indicate random hardware corruption. In that case it would be prudent to first backup all data you need (it is actually always a good idea to backup stuff). You could use scp/sftp to backup all important data to your workstation, or if you have some other server available make full system backup, for example:

otherserver# nc -l -p 8080 > brokenserver-backup.tar.gz
brokenserver# tar zcf - / | nc otherserver.example.com 8080

After having a working backup, you need to get a newer e2fsck as @G-Man suggested. However, as you discovered, you need statically compiled/linked version of e2fsck, or it will throw error about missing dynamic libraries. Here is how to do it.

To do that, first you need to download static version. Good thing about that is that it may be for any GNU/Linux distribution. I prefer ones from http://packages.debian.org/e2fsck-static :

su
cd /run
wget http://ftp.de.debian.org/debian/pool/main/e/e2fsprogs/e2fsck-static_1.44.2-1~bpo9+1_amd64.deb

Alternatively, you can download on your workstation and use sftp client like Filezilla to copy it to /run directory on broken server.

Then, you need to unpack it. For .deb, you use ar and tar.

brokenserver:/run# ar xv e2fsck-static_1.44.2-1~bpo9+1_amd64.deb 
x - debian-binary
x - control.tar.gz
x - data.tar.xz

brokenserver:/run# tar Jxvf data.tar.xz 
./
./sbin/
./sbin/e2fsck.static
./usr/
./usr/share/
./usr/share/doc/
./usr/share/doc/e2fsck-static/
./usr/share/doc/e2fsck-static/changelog.Debian.gz
./usr/share/doc/e2fsck-static/copyright
./usr/share/man/
./usr/share/man/man8/
./usr/share/man/man8/e2fsck.static.8.gz

Note: if ar unpacked data.tar.gz instead of data.tar.xz, then you would use tar zxvf data.tar.gz to unpack. Also, if you miss some of the tools (ar, tar, gzip, xz) or you run out of space, you can unpack this on any other GNU/Linux server or workstation and copy just the sbin/e2fsck.static file to brokenserver.

then, you finally get to fsck:

brokenserver:/run# chmod 755 /run/sbin/e2fsck.static
brokenserver:/run# ./sbin/e2fsck.static /dev/vda

Note: if you get "Permission denied" error, it is probably either because your /run is mounted noexec (not in your case, but it were, you'd fix it with mount -oremount,exec /run, or because you downloaded wrong architecture (for example, amd64 binary on old i386 - see arch(1)). strace(1) would give you more info at the end, like:

execve("/run/sbin/e2fsck.static", ["/run/sbin/e2fsck.static"], [/* 22 vars */]) = -1 ENOEXEC (Exec format error)

or

execve("/run/sbin/e2fsck.static", ["/run/sbin/e2fsck.static"], [/* 22 vars */]) = -1 EACCES (Permission denied)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.