xargs: argument line too long

Question

When I run below line in a shell script:

/sbin/iptables-save -t filter | 
grep -- "-A INPUT" | 
grep -v "fail2ban-\|f2b-" | 
sed -e "s#^-A#apply_rule /sbin/iptables -D#g" | 
xargs -0 echo -e "`declare -f apply_rule`\n" | 
/bin/bash

All it does is checking

-A INPUT -s IP -j DROP 

rule in iptables-save.

Say,

-A INPUT -s 198.55.114.215 -j DROP

Then convert it to Delete version and pass to xargs

apply_rule /sbin/iptables -D INPUT -s 198.55.114.215 -j DROP

But I have 2000 -A rules in iptables-save and when I try to run it, above line throws an error:

xargs: argument line too long

​I checked limits and

echo $(( $getconf ARG_MAX

result is:

2621440

This error does not occur when there 1000 -A rules, but 2000 -A rules in iptables-save creates this issue.

Actual error is line is:

xargs -0 echo -e "`declare -f apply_rule`\n"

How can I fix this issue?

Answer 1

Let's look at the pipeline:

/sbin/iptables-save -t filter | 
grep -- "-A INPUT" | 
grep -v "fail2ban-\|f2b-" | 
sed -e "s#^-A#apply_rule /sbin/iptables -D#g" | 
xargs -0 echo -e "`declare -f apply_rule`\n" | 
/bin/bash

iptables generates a list of rules, one rule per line. In other words, each rule is separated by a newline character, \n. With the options as shown, the grep and sed commands are processing their input one line at a time. In other words, they are also expecting newline-separated input and producing newline-separated output. xargs -0 however is expecting nul-separated input. Since the output of the commands that preceded contain no nul-characters, xargs tries to read all of its stdin in at once as a single item. That is why it generates the error message "argument line too long".

The solution is to tell xargs to expect newline-separated input. To do that, we add the option -d '\n'. However, we also want it to process only one line at a time. To do that, we need to specify -n1. Putting this all together:

/sbin/iptables-save -t filter | 
grep -- "-A INPUT" | 
grep -v "fail2ban-\|f2b-" | 
sed -e "s#^-A#apply_rule /sbin/iptables -D#g" | 
xargs -n1 -d '\n' echo -e "`declare -f apply_rule`\n" | 
/bin/bash

Documentation

From man xargs:

-d delim
Input items are terminated by the specified character. Quotes and backslash are not special; every character in the input is taken literally. Disables the end-of-file string, which is treated like any other argument. This can be used when the input consists of simply newline-separated items, although it is almost always better to design your program to use --null where this is possible. The specified delimiter may be a single character, a C-style character escape such as \n, or an octal or hexadecimal escape code. Octal and hexadecimal escape codes are understood as for the printf command. Multibyte characters are not supported.

-n max-args
Use at most max-args arguments per command line. Fewer than max-args arguments will be used if the size (see the -s option) is exceeded, unless the -x option is given, in which case xargs will exit.