I'm surprised that there isn't a definitive answer here. Someone should do some historical spelunking and figure this one out. Where and when did the "minus" or "dash" traditions start? Good thesis topic :-)
I picked up saying "minus" from a bunch of kernel hackers I worked with at a certain company that had an OS that included major parts of BSD (that would be apple). I always found that it tripped off the tongue much more easily than dash.
If I have a file name with a "-" in it I would never call it minus, I would call it dash. Thus, I can easily differentiate in conversation between the arguments part of a command and the file name part.
It's pretty rare to include arithmetical expressions in shell commands, so confusion with math seems unlikely.
Based on the other answers here, it sounds like people who are old-time UNIX gurus, or like myself have hung around with old-time UNIX gurus, are more like to say "minus". Thus my suspicion that there's an interesting historical story here.