After migrating a maildir network share (CIFS) to a new storage server, I started having problems with file permissions (users unable to delete old mail). My IMAP service is Dovecot but we can identify the problem at a lower level than that.
root I can navigate to a mail directory exhibiting this problem both on the server hosting the files and the server where they're mounted (hereafter referred as 'client'), and experience/observe the following behavior:
lsshows the same file listing and permissions on both machines.
- Permissions did not match exactly since the migration, but no permission changes affect the behavior. (All future operations are performed against files with permissions
-rw-rw---- 1 mail mail.)
- Performing operations as
rootalso has no effect.
Attempting permission changes such as
chmod g-rw *:
- on the server works fine
- on the client works fine for files created since the migration
on the client produces this error for pre-migration files:
chmod: changing permissions of '1479603582.M874812P11259.Pantheon,S=84750,W=85933:2,Sa': Invalid argument
- Attempting to read the file:
- on the server works fine
- on the client filename autocompletes and vim reports
Looking for any ACLs that might be affecting older files, I get this output on the server:
root@<storage-server>:/<path-to-share>/<site>/<user>/folders/cur# getfacl <filename> # file: <filename> # owner: mail # group: mail user::rw- group::rw- other::---
getfattr outputs nothing.
The storage server was previously an abandonware solaris+debian based OS (Nexenta) serving with CIFS on a ZFS storage pool. Now it is Ubuntu 16.04, again serving with CIFS on a ZFS storage pool. In all cases ACLs are/were supported but not used anywhere.
My Samba share config:
[Maildir] path = /<path-to-share> browseable = no guest ok = no valid users = mail writable = yes create mode = 0660 directory mode = 0770
which is mounted as
//<host>/Maildir /var/mail cifs auto,credentials=/root/.smb_mail,user,rw,exec 0 0. All zfs properties are at defaults for this filesystem and its parents.
I've tried removing create mode/directory mode, and adding @mail to valid users, all to no effect.
How else could my permissions be going wrong?
I tried switching to an NFS share instead, and the problem persists. I tried copying the maildir contents to a new location (in a different new filesystem) with
cp -r and
chown -r mail:mail, and again it persists, while served from a completely different path.
Finally, I tried
mv somefile backup && rm somefile && cat backup > somefile && chown mail:mail somefile, and attempting to read that file still fails with Permission Denied.
I'm at a loss as to how operations on specific files can be blocked in a manner that is independent of the sharing mechanism, logical location, unix permissions/ownership, and even any form of file metadata.
I had another go at switching to an NFS share and this time the permission issue went away. However I do not want to switch to NFS as it is causing other issues particularly for bootup. The issue is definitely with samba, but clearing all operational data (various .tdb and .dat files, etc.) also did not help.
The problem has been narrowed down to filenames with colons in them. Renaming a file to remove the colon renders it readable in the client, and renaming to an arbitrary, original name with a colon renders it unreadable. It appears Dovecot is renaming files over time to track some information and that adds a colon which eventually renders all mail unreadable/unwriteable, but this would have previously been the case as well.
Some additional observations: creating and then reading a file with a colon on the client works (and the colon appears as a double-quote on the server). Incidentally this is how we'd be getting filenames with colons in the first place... Newer files appear to have 2 colons, but on the server show as having one double quote and one colon.
It's starting to look like some kind of encoding issue - especially odd since these two systems are homogenous for the first time.