5

Example:

% touch -- safe-name -name-with-dash-prefix "name with space" \
    'name-with-double-quote"' "name-with-single-quote'" \
    'name-with-backslash\'

xargs can't seem to handle double quotes:

% ls | xargs ls -l 
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
ls: invalid option -- 'e'
Try 'ls --help' for more information.

If we use the -0 option, it has trouble with name that has dash prefix:

% ls -- * | xargs -0 -- ls -l --
ls: invalid option -- 'e'
Try 'ls --help' for more information.

This is before using other potentially problematic characters like newline, control character, etc.

  • Are you asking why ls doesn't work here? That's normal behavior, and that's why you never parse the output of ls. Is that your question or is there something else? – terdon Aug 11 '17 at 12:20
  • @terdon the first ls is just to supply the names, one line per entry. This is not about parsing the output of ls (because in this case the first ls just gives the filenames, without any other modification except adding newline per entry). The second ls is to demonstrate how xargs feeds the names as arguments to the command. In this case, xargs somehow fails to pass the second -- to ls. – Gerry Lufwansa Aug 11 '17 at 12:26
  • Not sure how you got that error message upon ls -- * | xargs -0 -- ls -l --. I'd expect ls be passed just one big argument (possibly starting with a - but that wouldn't be a problem with --) since the input doesn't contain NULs and the error would be about that file with a long name and some newline characters not being found. What system and version of ls/xargs is that? – Stéphane Chazelas Aug 11 '17 at 14:42
  • Like ls: cannot access '-name-with-dash-prefix'$'\n''name with space'$'\n''name-with-backslash\'$'\n''name-with-double-quote"'$'\n''name-with-single-quote'\'''$'\n''safe-name'$'\n': No such file or directory – Stéphane Chazelas Aug 11 '17 at 14:44
  • @StéphaneChazelas I'm on Debian 8.0 Jessie, xargs 4.4.2, ls 8.23. – Gerry Lufwansa Aug 11 '17 at 14:51
6

The POSIX specification does give you an example for that:

ls | sed -e 's/"/"\\""/g' -e 's/.*/"&"/' | xargs -E '' printf '<%s>\n'

(with filenames being arbitrary sequences of bytes (other than / and NULL) and sed/xargs expecting text, you'd also need to fix the locale to C (where all non-NUL bytes would make valid characters) to make that reliable (except for xargs implementations that have a very low limit on the maximum length of an argument))

The -E '' is needed for some xargs implementations that without it, would understand a _ argument to signify the end of input (where echo a _ b | xargs outputs a only for instance).

With GNU xargs, you can use:

ls | xargs -d '\n' printf '<%s>\n'

GNU xargs has also a -0 that has been copied by a few other implementations, so:

ls | tr '\n' '\0' | xargs -0 printf '<%s>\n'

is slightly more portable.

All of those assume the file names don't contain newline characters. If there may be filenames with newline characters, the output of ls is simply not post-processable. If you get:

a
b

That can be either two a and b files or a file called a<newline>b, there's no way to tell.

GNU ls has a --quoting-style=shell-always which makes its output unambiguous and could be post-processable, but the quoting is not compatible with the quoting expected by xargs. xargs recognise "...", \x and '...' forms of quoting. But both "..." and '...' are strong quotes and can't contain newline characters (only \ can escape newline characters for xargs), so that's not compatible with sh quoting where only '...' are strong quotes (and can contain newline characters) but \<newline> is a line-continuation (is removed) instead of an escaped newline.

You can use the shell to parse that output and then output it in a format expected by xargs:

eval "files=($(ls --quoting-style=shell-always))"
[ "${#files[#]}" -eq 0 ] || printf '%s\0' "${files[@]}" |
  xargs -0 printf '<%s>\n'
  • So to recap it's quite simple actually (I should've read the manpage once in a while :-) ): xargs interprets quotes (and backslash) except when given -0 in which it only interprets NUL as record separator. Thanks for explanation about the different flavors of xargs. – Gerry Lufwansa Aug 12 '17 at 0:51
  • Beside / and NUL, the link you quote also names <new-line> and {} as failures. – Arrow Aug 14 '17 at 1:41
  • @Arrow, the example there is using -I {} and $1, $2 as arguments to xargs, so if {} is found in $1 and $2 in that case, it's a problem. But not here as we're neither using -I nor passing variable arguments to xargs. Newline is covered in my answer. I should add -E '' though – Stéphane Chazelas Aug 14 '17 at 6:30
3

For xargs to understand the -0 null-delimited input option, the sending party must also apply the null delimiter to the data that they are sending over.

Else there's no synchronization between the two.

One option is the GNU find command which can place such delimiters:

find . -maxdepth 1 ! -name . -print0 | xargs -0 ls -ld
3

As you said, xargs doesn't like unmatched double quotes unless you use -0 but -0 only makes sense if you feed it null-terminated data. So, this fails:

$ echo * | xargs
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
name-with-backslash -name-with-dash-prefix

But this works:

$ printf '%s\0' -- * | xargs -0
-- name-with-backslash\ -name-with-dash-prefix name-with-double-quote" name-with-single-quote' name with space safe-name

In any case, your basic approach is not really the best way to do this. Instead of fiddling about with xargs and ls and whatnot, just use shell globs instead:

$ for f in *; do ls -l -- "$f"; done
-rw-r--r-- 1 terdon terdon 4142 Aug 11 16:03 a
-rw-r--r-- 1 terdon terdon 0 Aug 11 15:34 'name-with-backslash\'
-rw-r--r-- 1 terdon terdon 0 Aug 11 15:34 -name-with-dash-prefix
-rw-r--r-- 1 terdon terdon 0 Aug 11 15:34 'name-with-double-quote"'
-rw-r--r-- 1 terdon terdon 0 Aug 11 15:34 "name-with-single-quote'"
-rw-r--r-- 1 terdon terdon 0 Aug 11 15:34 'name with space'
-rw-r--r-- 1 terdon terdon 0 Aug 11 15:34 safe-name
  • "In any case, your basic approach is wrong to begin with. Instead of fiddling about with xargs and ls and whatnot, just use shell globs instead". i should've added that I specifically asked about solution that involves xargs, because the names do not necessarily come from filenames, and I want to pass multiple arguments to a single command, not one argument to one command. – Gerry Lufwansa Aug 11 '17 at 14:00
  • "And there's no need to use the -- with ls when run via xargs, xargs can deal with that properly itself:" it seems that -- is actually needed, e.g.: ls -- *dash* | xargs ls result in error while ls -- *dash* | xargs ls -- is okay. xargs does not automatically add an extra -- argument and the second ls needs it to separate the name from options. – Gerry Lufwansa Aug 11 '17 at 14:03
  • @GerryLufwansa ah! That's a very different proposition then, yes. You can probably still use the shell loop (personally, I think I never use xargs and always write such loops), but we might be abe to give you a better solution for your use case if you clarify what you'll be doing. – terdon Aug 11 '17 at 14:03
  • @GerryLufwansa huh. You're absolutely right. That's odd, I guess xargs only does that when run with -0. Thanks for the correction. – terdon Aug 11 '17 at 14:04
0

It is extremelly silly to try to parse the output of a command ls that is not designed to be parsed to feed a command which is not designed to deal with several characters (for example: new lines and {}) when the shell does that by itself:

set -- *; for f; do echo "<$f>"; done

set    -- *
for    f
do     ls "$f"
done

Or, in one command line:

$ set -- *; for f; do echo "<$f>"; done
<name-with-backslash\>
<-name-with-dash-prefix>
<name-with-double-quote">
<name-with-single-quote'>
<name with space>
<safe-name>
<with_a
newline>

Note that the output deals (and has n example as last filename) with new-lines perfectly fine.

Or, if the number of files makes the shell slow, use find:

$ find ./ -type f -exec echo '<{}>' \;
<./safe-name>
<./with_a
newline>
<./name-with-double-quote">
<./-name-with-dash-prefix>
<./name with space>
<./name-with-single-quote'>
<./name-with-backslash\>

Just mind that find process all dot-files and all sub-directories diferently than the shell.

  • If only find can feed multiple filenames to a single command, so we don't need xargs, that'd be nice. – Gerry Lufwansa Aug 16 '17 at 11:59
  • And indeed, not only can find use multiple commands, but it can execute multiple commands with the same argument using the -exec sh concept and here. @GerryLufwansa – Arrow Aug 16 '17 at 15:11
  • BTW, in the question ls is just an example. This is not about whether we should or should not be parsing the output of ls. What I asked is how to handle characters like double quote or newline in xargs, and the answer provided by others is to use -0 and to feed it null-separated names instead of newline-separated. Your answer is irrelevant in this case because I was asking specifically about xargs. – Gerry Lufwansa Aug 17 '17 at 7:32
  • About find, what I commented was about feeding multiple arguments to a single command, not about using/running multiple commands. – Gerry Lufwansa Aug 17 '17 at 7:34
  • And lastly, using find -exec is sometimes inadequate. When we want to limit the number of results, we must feed the output of find to head or tail first, as find cannot limit number of results by itself. When we want to sort the results e.g. by mtime, size, etc we need to use other tools because find does not do sorting. – Gerry Lufwansa Aug 17 '17 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.