2

I have a file input that may have blank lines, I want any empty lines skipped.
I am trying to find a solution that does not require using grep / tr and piping the output to xargs

I use the following xarg options

-r in the event of no arguments I do not want the command run. -d '\n' new line is the delimiter.

command using echo:

echo "" | xargs -r -d'\n' -I {}  echo "test '{}'"

output:

test ''

command using file made of empty lines:

xargs -a /tmp/test.txt -r -d'\n' -I {}  echo "test '{}'"

output:

test ''
test ''
test ''

I could use grep as mentioned, but Id like to know if I can do it with just xargs.

grep example (working):

grep -v -e '^$' /tmp/test.txt | xargs -r -d'\n' -I {}  echo "test '{}'"
5
  • 1
    Try it without the -0 option to xargs.
    – Sotto Voce
    Sep 4, 2022 at 7:15
  • as mentioned the -0 option is required. some lines will have a single quote that cannot be escaped. Sep 4, 2022 at 7:20
  • From the part of the man page that discusses the use of -0: When using this option you will need to ensure that the program which produces the input for xargs also uses a null character as a separator. Your echo and grep commands don't output a null character at the end of each line. (a null character is a distinct character, not the same thing as an empty string or partial string)
    – Sotto Voce
    Sep 4, 2022 at 7:39
  • as mentioned, the actual input is a file, the echo is serving as a test. Sep 4, 2022 at 7:55
  • @SottoVoce you were correct I should not have been using -0. It conflicts with -n which also will not escape characters and was also overriding -0 anyway. Thanks for raising that. Sep 4, 2022 at 13:51

3 Answers 3

8

-r is not to ignore empty elements, it is to avoid running the command once if the input has no element (it's not needed with -I (nor BSD's -J) though).

You generally want -r for that reason.

find . -criteria -print0 | xargs -0 ls -ld

If find didn't find anything would still run ls -d without argument, and would list ..

So you do want:

find . -criteria -print0 | xargs -r0 ls -ld

For ls not to be run at all if find finds nothing.

Note that both -r and -0 are GNU extensions (same for -d which is even less portable). -r is the default behaviour in the xargs of some BSDs including FreeBSD's and NetBSD's (though that makes them non-POSIX compliant).

(in any case, no need for xargs in that particular example as you can do find . -criteria -exec ls -ld {} +).

As far as I know, GNU xargs is the only implementation that supports a -d option, and it has no support for filtering out specific argument values, be that empty ones or otherwise, so using grep (grep -v '^$' or LC_ALL=C grep .) is the way to go.

If you still wanted to use -a (another GNU extension), for instance so as to leave stdin untouched for the command, and with a shell with support for ksh-style process substitution (ksh, zsh, bash), you could do:

xargs -rd '\n' -a <(grep -v '^$' test.txt) cmd --
xargs -d '\n' -I{} -a <(grep -v '^$' test.txt) cmd -- {}

(the rc, akanga, es, fish and yash shells have equivalent features with different syntax).

Note that -0 is like -d '\0', so conflicts with -d '\n'. You have to choose which delimiter you want. The latter will take precedence in -0 -d '\n' as it comes last.

-0 is the preferred delimiter as that's the one byte value that cannot occur in a command's argument or a file's path. -0, contrary to -d is now found in many other xargs implementations (including those of most BSDs, Solaris, busybox, toybox, ast-open). With xargs implementations that don't support -d (that is, all but GNU xargs), you can use:

tr '\n' '\0' | xargs -0 ...

In place of GNU's:

xargs -d '\n' ...

If using zsh, instead of using xargs, you could do:

for arg ( ${(f)"$(<test.txt)"} ) cmd -- $arg

The f parameter expansion flag splits on newlines, and as ${...} is left unquoted, empty elements are removed.

With bash:

readarray -t args < test.txt &&
  for arg in "${args[@]}"; do
    [ -z "$arg" ] || cmd -- "$arg"
  done

You could also do empty removal there by leaving the array expansion unquoted, but you'd also need to disable globbing and splitting which are also done upon unquoted parameter expansions:

IFS=; set -o noglob
readarray -t args < test.txt &&
  for arg in ${args[@]}; do
    cmd -- "$arg"
  done

You can also do things like:

while IFS= read <&3 -r arg || [ -n "$arg" ]; do
  [ -z "$arg" ] || cmd -- "$arg" 3<&-
done 3< test.txt

Which is standard sh syntax.

7
  • Im not using find. Sep 4, 2022 at 7:58
  • @JacobJarick, I'm just mentioning find as an example for when you want -r (most of the times you want it). Sep 4, 2022 at 7:59
  • thanks, so it does seem the only way is grep the file and pipe to xargs. Sep 4, 2022 at 8:01
  • thanks I appreciate your clarification on the -r option. I interpreted it as a per argument input (eg some lines being empty) not as a complete lack of inputs (no arguments at all). Sep 4, 2022 at 8:29
  • 1
    @JacobJarick, -0 is to process NUL-delimited records as output by find -print0 for instance. printf 'a\nb\0c\0' produces 2 of those records, the first one containing a<newline>b and the second containing c. Sep 4, 2022 at 12:30
4

-0 I don't want ' and other characters escaped.
-r I want blank lines ignored.
-d '\n' new line is the delimiter.

The combination of -0 and -d '\n' doesn't seem to make sense to me. -0 tells xargs to use the NUL byte as a delimiter, while -d '\n' tells it to use a newline as the delimiter. Both disable quote and backslash processing.

-0, --null
Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (every character is taken literally). [...]

--delimiter=delim, -d delim

Input items are terminated by the specified character. [...] When processing the input, quotes and backslash are not special; every character in the input is taken literally. [...]

With the version of GNU find I have, the latter one seems to apply if both are given, compare e.g.:

$ printf 'foo\nbar\n' | xargs -0 -d'\n' -I {}  echo "<{}>"
<foo>
<bar>
$ printf 'foo\nbar\n' | xargs -d'\n' -0 -I {}  echo "<{}>"
<foo
bar
>

If you have an input with empty elements, and want to use xargs, I would just pipe through grep ., or grep -z . for NUL-separated values. (Or grep -v '^$' if you need to accept lines that only contain byte sequences that don't form valid characters in the current locale.)

Filtering lines is what grep is for anyway, and since you're already forking off xargs as a separate process, adding the grep doesn't hurt much. Also it's easy to modify the regex to e.g. drop lines with only whitespace, if that's ever required.

2
  • I will edit my question as -0 and -d are conflicting options where the later takes precedence. this was pointed out to me by Stéphane Chazelas Sep 4, 2022 at 13:43
  • 3
    Beware printf '\200\201\202\n' | grep . for instance filters out that non-empty line in a locale using UTF-8 as charmap for instance. You'd need LC_ALL=C grep . for that to be equivalent to grep -v '^$' Sep 4, 2022 at 14:35
2

Answering my own question to confirm you must use grep (or another tool) and pipe ouput to xargs to filter out blank inputs.

Solution:

grep -v '^$' /tmp/test.txt | xargs -r -d'\n' -I {}  echo "test '{}'"

As clarified by Stéphane Chazelas -r does not filter empty inputs but prevents running if there are no inputs.

from the xargs man page -r, --no-run-if-empty section:
Normally, the command is run once even if there is no input.

Additional note gained from further edits, -0 and -d are conflicting arguments with the later taking precedence.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .