ACL not working on directories

Question

For some reason when I create new file from Windows I can change the permissions just fine but when I create a folder it won't let me change the permissions, I get this error:

A device attached to the system is not functioning.

I'm running AIX 6.1 and Samba 3.6. Here's how my smb.conf look like:

[global]
        .....
        .....
        nt acl support = yes
        inherit acls = Yes
        map acl inherit = Yes
        map archive = no
        map hidden = no
        map read only = no
        map system = no
        store dos attributes = yes
        inherit permissions = Yes

[testshare]
        comment = Test
        path = /test
        writable = yes
        read only = no
        force group = "Domain Admins"
        valid users = @"Domain Admins"
        directory mode = 0770
        force directory mode = 0770
        create mode = 0660
        force create mode = 0660
        access based share enum = yes
        hide unreadable = yes

Here's the acl for the filesystem and umask:

# aclget /test
*
* ACL_type   AIXC
*
attributes: 
base permissions
    owner(administrator):  rwx
    group(domain^admins):  rwx
    others:  rwx
extended permissions
    disabled

# umask -S
u=rwx,g=rx,o=rx

I really don't understand why I can set file permissions (from Windows) but not directories.

Answer 1

The (in 1997) withdrawn POSIX draft ACLs are not able to support the features you get from the Microsoft CIFS.

Given that AIX supports modern NFSv4 ACLs that have been modeled after NTFS ACLs, you may like to look at the NFSv4 ACLs.

But since you are using Samba, it may be that the basic problem is that Samba does not yet support NFSv4/NTFS ACLs on the underlying filesystem.

There is a patch from Oracle for Solaris that fixes this problem and there may be a patch from IBM as well.