2

After migrating a maildir network share (CIFS) to a new storage server, I started having problems with file permissions (users unable to delete old mail). My IMAP service is Dovecot but we can identify the problem at a lower level than that.

As root I can navigate to a mail directory exhibiting this problem both on the server hosting the files and the server where they're mounted (hereafter referred as 'client'), and experience/observe the following behavior:

  • ls shows the same file listing and permissions on both machines.
  • Permissions did not match exactly since the migration, but no permission changes affect the behavior. (All future operations are performed against files with permissions -rw-rw---- 1 mail mail.)
  • Performing operations as mail instead of root also has no effect.
  • Attempting permission changes such as chmod g-rw *:

    • on the server works fine
    • on the client works fine for files created since the migration
    • on the client produces this error for pre-migration files:

      chmod: changing permissions of '1479603582.M874812P11259.Pantheon,S=84750,W=85933:2,Sa': Invalid argument

  • Attempting to read the file:
    • on the server works fine
    • on the client filename autocompletes and vim reports Permission Denied

Looking for any ACLs that might be affecting older files, I get this output on the server:

root@<storage-server>:/<path-to-share>/<site>/<user>/folders/cur# getfacl <filename>
# file: <filename>
# owner: mail
# group: mail
user::rw-
group::rw-
other::---

getfattr outputs nothing.

The storage server was previously an abandonware solaris+debian based OS (Nexenta) serving with CIFS on a ZFS storage pool. Now it is Ubuntu 16.04, again serving with CIFS on a ZFS storage pool. In all cases ACLs are/were supported but not used anywhere.

My Samba share config:

[Maildir]
    path = /<path-to-share>
    browseable = no
    guest ok = no
    valid users = mail
    writable = yes
    create mode = 0660
    directory mode = 0770

which is mounted as //<host>/Maildir /var/mail cifs auto,credentials=/root/.smb_mail,user,rw,exec 0 0. All zfs properties are at defaults for this filesystem and its parents.

I've tried removing create mode/directory mode, and adding @mail to valid users, all to no effect.

How else could my permissions be going wrong?

Update:

I tried switching to an NFS share instead, and the problem persists. I tried copying the maildir contents to a new location (in a different new filesystem) with cp -r and chown -r mail:mail, and again it persists, while served from a completely different path.

Finally, I tried mv somefile backup && rm somefile && cat backup > somefile && chown mail:mail somefile, and attempting to read that file still fails with Permission Denied.

I'm at a loss as to how operations on specific files can be blocked in a manner that is independent of the sharing mechanism, logical location, unix permissions/ownership, and even any form of file metadata.

Update 2:

I had another go at switching to an NFS share and this time the permission issue went away. However I do not want to switch to NFS as it is causing other issues particularly for bootup. The issue is definitely with samba, but clearing all operational data (various .tdb and .dat files, etc.) also did not help.

Update 3:

The problem has been narrowed down to filenames with colons in them. Renaming a file to remove the colon renders it readable in the client, and renaming to an arbitrary, original name with a colon renders it unreadable. It appears Dovecot is renaming files over time to track some information and that adds a colon which eventually renders all mail unreadable/unwriteable, but this would have previously been the case as well.

Some additional observations: creating and then reading a file with a colon on the client works (and the colon appears as a double-quote on the server). Incidentally this is how we'd be getting filenames with colons in the first place... Newer files appear to have 2 colons, but on the server show as having one double quote and one colon.

It's starting to look like some kind of encoding issue - especially odd since these two systems are homogenous for the first time.

1

Samba does not allow colons in filenames and uses character remapping (to quotes) to support them in a presumably windows-friendly way. At some point the handling of this was different, resulting in actual colons in filenames server-side. That, or I may at some point have actually copied the files via the share. (Unlikely, as everything was maintained with zfs incremental sends during the transition.) Additionally, when using an NFS server, Dovecot started applying renames that broke Samba for newer content as well.

Since the colons are part of disposable metadata embedded in the filename which gets regenerated if incorrect, I used a script to strip all real colons from the share: find "$@" -name '*:*' -exec rename 's/://g' {} +.

(I tried being a bit more clever but the filename only appears to have a double quote in place of colon, and working out the precise translation is of minimal value for the effort, especially after 12 hours of head banging.)

After this all files became readable again, and the only real data loss was having to mark a bunch of emails read again. I only hope this fix is a one-time affair.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.