xargs ignore blank line when using -d'\n'

Question

I have a file input that may have blank lines, I want any empty lines skipped.
I am trying to find a solution that does not require using grep / tr and piping the output to xargs

I use the following xarg options

-r in the event of no arguments I do not want the command run. -d '\n' new line is the delimiter.

command using echo:

echo "" | xargs -r -d'\n' -I {}  echo "test '{}'"

output:

test ''

command using file made of empty lines:

xargs -a /tmp/test.txt -r -d'\n' -I {}  echo "test '{}'"

output:

test ''
test ''
test ''

I could use grep as mentioned, but Id like to know if I can do it with just xargs.

grep example (working):

grep -v -e '^$' /tmp/test.txt | xargs -r -d'\n' -I {}  echo "test '{}'"

Answer 1

-r is not to ignore empty elements, it is to avoid running the command once if the input has no element (it's not needed with -I (nor BSD's -J) though).

You generally want -r for that reason.

find . -criteria -print0 | xargs -0 ls -ld

If find didn't find anything would still run ls -d without argument, and would list ..

So you do want:

find . -criteria -print0 | xargs -r0 ls -ld

For ls not to be run at all if find finds nothing.

Note that both -r and -0 are GNU extensions (same for -d which is even less portable). -r is the default behaviour in the xargs of some BSDs including FreeBSD's and NetBSD's (though that makes them non-POSIX compliant).

(in any case, no need for xargs in that particular example as you can do find . -criteria -exec ls -ld {} +).

As far as I know, GNU xargs is the only implementation that supports a -d option, and it has no support for filtering out specific argument values, be that empty ones or otherwise, so using grep (grep -v '^$' or LC_ALL=C grep .) is the way to go.

If you still wanted to use -a (another GNU extension), for instance so as to leave stdin untouched for the command, and with a shell with support for ksh-style process substitution (ksh, zsh, bash), you could do:

xargs -rd '\n' -a <(grep -v '^$' test.txt) cmd --

xargs -d '\n' -I{} -a <(grep -v '^$' test.txt) cmd -- {}

(the rc, akanga, es, fish and yash shells have equivalent features with different syntax).

Note that -0 is like -d '\0', so conflicts with -d '\n'. You have to choose which delimiter you want. The latter will take precedence in -0 -d '\n' as it comes last.

-0 is the preferred delimiter as that's the one byte value that cannot occur in a command's argument or a file's path. -0, contrary to -d is now found in many other xargs implementations (including those of most BSDs, Solaris, busybox, toybox, ast-open). With xargs implementations that don't support -d (that is, all but GNU xargs), you can use:

tr '\n' '\0' | xargs -0 ...

In place of GNU's:

xargs -d '\n' ...

If using zsh, instead of using xargs, you could do:

for arg ( ${(f)"$(<test.txt)"} ) cmd -- $arg

The f parameter expansion flag splits on newlines, and as ${...} is left unquoted, empty elements are removed.

With bash:

readarray -t args < test.txt &&
  for arg in "${args[@]}"; do
    [ -z "$arg" ] || cmd -- "$arg"
  done

You could also do empty removal there by leaving the array expansion unquoted, but you'd also need to disable globbing and splitting which are also done upon unquoted parameter expansions:

IFS=; set -o noglob
readarray -t args < test.txt &&
  for arg in ${args[@]}; do
    cmd -- "$arg"
  done

You can also do things like:

while IFS= read <&3 -r arg || [ -n "$arg" ]; do
  [ -z "$arg" ] || cmd -- "$arg" 3<&-
done 3< test.txt

Which is standard sh syntax.

Answer 2

-0 I don't want ' and other characters escaped.
-r I want blank lines ignored.
-d '\n' new line is the delimiter.

The combination of -0 and -d '\n' doesn't seem to make sense to me. -0 tells xargs to use the NUL byte as a delimiter, while -d '\n' tells it to use a newline as the delimiter. Both disable quote and backslash processing.

-0, --null
Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (every character is taken literally). [...]

--delimiter=delim, -d delim

Input items are terminated by the specified character. [...] When processing the input, quotes and backslash are not special; every character in the input is taken literally. [...]

With the version of GNU find I have, the latter one seems to apply if both are given, compare e.g.:

$ printf 'foo\nbar\n' | xargs -0 -d'\n' -I {}  echo "<{}>"
<foo>
<bar>
$ printf 'foo\nbar\n' | xargs -d'\n' -0 -I {}  echo "<{}>"
<foo
bar
>

If you have an input with empty elements, and want to use xargs, I would just pipe through grep ., or grep -z . for NUL-separated values. (Or grep -v '^$' if you need to accept lines that only contain byte sequences that don't form valid characters in the current locale.)

Filtering lines is what grep is for anyway, and since you're already forking off xargs as a separate process, adding the grep doesn't hurt much. Also it's easy to modify the regex to e.g. drop lines with only whitespace, if that's ever required.

Answer 3

Answering my own question to confirm you must use grep (or another tool) and pipe ouput to xargs to filter out blank inputs.

Solution:

grep -v '^$' /tmp/test.txt | xargs -r -d'\n' -I {}  echo "test '{}'"

As clarified by Stéphane Chazelas -r does not filter empty inputs but prevents running if there are no inputs.

from the xargs man page -r, --no-run-if-empty section:
Normally, the command is run once even if there is no input.

Additional note gained from further edits, -0 and -d are conflicting arguments with the later taking precedence.