system("cmd") function typically forks a process, and in the child process, runs the system's shell (typically
/bin/sh) with as arguments,
["sh", "-c", "cmd"], to have
sh parse and execute that shell command line.
As an optimisation, it may sometimes do without a shell call, if
cmd doesn't contain any shell meta-characters (like quoting characters or globbing characters or things like
&&...) other than space and tab, but here we have shell meta-characters since we have a
ls -U -1 dir/* will be interpreted by the system's shell.
It is the shell that expands
dir/* to a list of matching files passed to
ls, so the way it's done depends on the shell.
In a terminal, you typically run your login shell, which is generally not
/bin/sh. Also (as noted by peterph), that shell, since it's run interactively will typically read configuration files like
~/.zshrc (if the shell is
zsh) where some settings might affect how globbing is done.
My shell is
zsh, and I've got a
setopt dotglob in my
$ echo *
.a d é f
Without reading the
$ zsh -c 'echo *'
d é f
$ LC_ALL=C zsh -c 'echo *'
d f é
You'll notice that
zsh honours the locale when sorting the list.
$ sh -c 'echo *'
d f é
sh (which in my case is Debian
ash) doesn't honour the locale and sorts as if in the C locale.
If you want
system() to interpret the command line with a particular shell, you can write it:
system("zsh", "-c", "cmd");
When passed more that one argument,
system() never implicitly calls a shell, so above, it forks a process in which it runs
["zsh", "-c", "cmd"] as arguments.