My first thought is that SMB is probably designed to work with the Windows model of security...
SMB = server message block. It is not SAMBA.
Server Message Block (SMB) is a communication protocol originally developed in 1983 by Barry A. Feigenbaum at IBM and intended to provide shared access to files and printers across nodes on a network of systems running IBM's OS/2. It also provides an authenticated inter-process communication (IPC) mechanism. In 1987, Microsoft and 3Com implemented SMB in LAN Manager for OS/2, at which time SMB used the NetBIOS service atop the NetBIOS Frames protocol as its underlying transport. Later, Microsoft implemented SMB in Windows NT 3.1 and has been updating it ever since, adapting it to work with newer underlying transports.
So SMB was not
designed work with a [Microsoft] Windows model of security; Server Message Block came long before Windows 3.1 in 1992, and windows95 in... 1995. It is a protocol that everyone began to use including Microsoft, and has been updated to SMB 2.0 and 3.0. Like everything back in the early days of computing, security concepts were largely non existent; to say Windows model of security is grossly ambiguous or non-specific.
Basically, I have Samba running on one Linux box, and I can mount the file share on another Linux box...
SAMBA is a linux software suite that provides interoperability to Microsoft Windows systems providing file & print services and to seamlessly integrate linux servers into Active Directory environments. Can you export a file share using SAMBA using SMB (or obsolete CIFS?) and have another linux server mount it, sure. But I would not do that, based on the simple definition and purpose what SAMBA is supposed to do. Between linux and linux my recommendation would be using NFS
vers=4.2. If you want to specifically use SAMBA making use of SMB3.0 then I would scour the samba.org website for all the configuration options in
/etc/samba/smb.conf to properly export and mount such a share to that linux applies its file system stuff (permissions, etc.) which today would/should be over the SMB3.0 protocol.
it appears to be impossible to turn off execute permission at all.
this makes me think of using an NTFS file system in linux...
if you are using
ntfs-3g in linux to mount an NTFS file system, instead of a linux supported file system such as XFS or EXT3/4, then you will always see execute permissions on everything, unless you explicitly mount it to not be so.
Do Linux file security settings work on SMB?
I was say yes, but from what you've described you have left out a lot of important context - Linux version being used, SAMBA version being used, your
smb.conf and the SMB protocol actually in effect, so I would guess that you have something misconfigured... you mentioned maybe somebody has extended the protocol to add support for that ... that answer would be in how SMB has been updated to it's latest version being 3.0. I have zero problem using linux samba export via 3.0 and a microsoft win10 accessing it, with samba
passdb backend = tdbsam.