6

I have a rw file system /myraid0.

# cat /proc/mounts | grep myraid0
/dev/mapper/isw_cfdbejjgdi_myraid0p1 /myraid0 ext4 rw,relatime,stripe=8,data=ordered 0 0

I have no problem writing to /myraid0 as any user.

redis@host:~$ echo hi > /myraid0/tmp/redis/test
redis@host:~$ cat /myraid0/tmp/redis/test
hi

However, my process can't write a file on /myraid0, due to EROFS (Read-only file system).

open("temp-4036.rdb", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EROFS (Read-only file system)

This shows the process views the mount as ro.

# cat /proc/15920/mounts | grep myraid0
/dev/mapper/isw_cfdbejjgdi_myraid0p1 /myraid0 ext4 ro,relatime,stripe=8,data=ordered 0 0

Why does the process only have read only view of the mount?

Thanks!

Additional details

  • Ubuntu 16.04
  • Linux jeff-apartment-2015 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • App is redis-server. It's failing to save the snapshots to a non-root device
  • Redis error is

Failed opening .rdb for saving: Read-only file system

  • Installed version 3.0.6 via apt-get
  • Launched by systemd
  • Listing of /proc/15920/ns (requested by @VenkatC): lrwxrwxrwx 1 redis redis 0 Sep 16 05:39 cgroup -> cgroup:[4026531835] lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 ipc -> ipc:[4026531839] lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 mnt -> mnt:[4026532343] lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 net -> net:[4026531957] lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 pid -> pid:[4026531836] lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 user -> user:[4026531837] lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 uts -> uts:[4026531838]
  • apparmor_status | grep redis returns nothing (requested by @Gilles)
  • Nothing else in the logs more informative than the strace output.

Process namespaces

Working bash shell for redis user

ls -l /proc/7359/ns/mnt
lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 /proc/7359/ns/mnt -> mnt:[4026531840]

Not working redis process

ls -l /proc/15920/ns/mnt
lrwxrwxrwx 1 redis redis 0 Sep 15 18:03 /proc/15920/ns/mnt -> mnt:[4026532343]

Working bash shell for my user

ls -atlrh /proc/7138/ns/mnt
lrwxrwxrwx 1 jeff jeff 0 Sep 15 18:03 /proc/7138/ns/mnt -> mnt:[4026531840]
  • Can you post listing of /proc/$$/ns (your shell) and /proc/15920/ns ? The process might be running in different mount namespace – VenkatC Sep 16 '16 at 1:38
  • Are you sure that the process is writing in the desired directory? Are there any security restrictions (e.g. SELinux, AppArmor, …)? Does anything appear in the system logs when the process fails to access that file? – Gilles Sep 16 '16 at 7:18
  • Updated question for @Gilles. – Jeff Sep 16 '16 at 12:53
  • Please post the full error redis generates (relevant source code is at github.com/antirez/redis/blob/…, line 958, suggests you should see error along lines of "Failed opening the RDB file %s (in server root dir %s) for saving: %s". – steve Sep 16 '16 at 13:03
  • Updated question for @VenkatC. It looks like it's in its own process namespace! – Jeff Sep 16 '16 at 13:06
10

As you can see, redis process is running in it's own 'mount namespace', with readonly option.

Redis proc started by Systemd [/proc/15920/ns/mnt] -> mnt:[4026532343]

your shell [/proc/7138/ns/mnt] -> mnt:[4026531840]

Look at systemd unit for the redis-server startup and update options related to the mountflags to suit your needs

Looking at redis-server systemd unit file, I see below settings

# grep -i readwrite /etc/systemd/system/redis.service 
ReadWriteDirectories=-/var/lib/redis
ReadWriteDirectories=-/var/log/redis
ReadWriteDirectories=-/var/run/redis
ReadWriteDirectories=-/etc/redis

so you could add /myraid0 as an additional ReadWriteDirectories and restart redis service

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.