8

Good evening,

The following is a piece of the code I'm using in a script. Launching from a SSH sesssion works fine, however, when it runs via cron, it displays broken pipe errors on screen.

I can't reproduce it via SSH.

Code:

IP=$(sort --random-sort /root/ips.csv | head -n 1); nc -zv -w 2 $IP 443 2>&1 | grep succeeded >> outfile

Error in screen:

sort: write failed: standard output; Broken pipe
sort: write error

Any tips/pointers?

Thank you!

3
  • 1
    Chances are that cron's current working directory is not the location of ips.csv, so the file is not there to read, breaking the pipe. Try IP=$(sort --random-sort /path/to/ips.csv [...].
    – DopeGhoti
    Mar 31, 2017 at 20:58
  • Good point. Actually I have the full path to the file, but I forgot to include it when editing the code to post online.
    – t988GF
    Mar 31, 2017 at 21:03
  • Since I can't seem to figure out how to make this code work, are there any alternatives to this code? Objective is to randomize a list and connect to the first IP (occurance) of that list?
    – t988GF
    Apr 1, 2017 at 21:31

3 Answers 3

11

When head finishes after handling the first line, it exits, closing the other end of the pipe. sort may still be trying to write more, and writing to a closed pipe or socket returns the EPIPE error. But it also raises the SIGPIPE signal, killing the process, unless the signal is ignored or handled. With the signal ignored, sort sees the error, complains, and exits. If the signal is not ignored, sort just dies.

We can use the trap builtin to ignore a signal from the shell, getting the error:

$ trap "" PIPE
$ sort bigfile | head -1 > /dev/null 
sort: write failed: standard output: Broken pipe
sort: write error

But sadly, we can't use trap to un-ignore the signal and get the desired behaviour, as POSIX requires that it's not allowed to do that in non-interactive shells (scripts). It does allow it for interactive shells, but Bash's trap doesn't do it in that case either.

To test:

sh$ trap '' PIPE                     # ignore the signal    
sh$ PS1='another$ ' bash             # run another shell
another$ trap - PIPE                 # try to reset the signal
                                     # it doesn't work
another$ sort bigfile |head -1 > /dev/null
sort: write failed: 'standard output': Broken pipe
sort: write error

Instead, we could use an external tool, like a Perl one-liner to run a script or command with the signal un-ignored (sort exits silently here):

another$ perl -e '$SIG{PIPE}="DEFAULT"; exec "@ARGV"' \
         'sort bigfile |head -1' > /dev/null 
another$ 

As for your situation with cron, the reason could be that systemd apparently makes SIGPIPE ignored by default, mentioning:

[SIGPIPE is] not really useful for normal daemons though, and as we try to provide a good, useful execution environment for daemons, we turn this off. Of course, shells and suchlike should turn this on again.

Of course, this is also mentioned in the documentation (systemd.exec):

IgnoreSIGPIPE=
Takes a boolean argument. If true, causes SIGPIPE to be ignored in the executed process. Defaults to true because SIGPIPE generally is useful only in shell pipelines.

On my Debian system, /lib/systemd/system/cron.service explicitly sets IgnoreSIGPIPE=false, undoing the systemd default for cron. You may want to check if that would help in your case.

0
1

Since --random-sort is a GNU extension anyway, you might as well use GNU shuf which comes from the same collection of GNU utilties (GNU coreutils):

ip=$(shuf -n 1 ips.tsv)

Beside avoiding the SIGPIPE issue, that is also more efficient as it doesn't need to shuffle the whole file as GNU sort --random-sort does (shuf uses some Reservoir Sampling algorithm on large inputs to do that).

0

As a general solution you could use this kind of Bash function to run pipe without broken pipe error and without hiding other possible error messages:

entire_pipe() {
    ( set -m; (
        trap 'exit 0' INT
        echo $BASHPID
        $1
    ) & wait $! ) |
    (
        trap 'trap - EXIT; kill -s INT -- -$pid 2>/dev/null || :' EXIT
        read -r pid
        $2
    )
}

To make this more portable, you could replace $BASHPID by $(exec sh -c 'echo "$PPID"').

To use the function in case of this question you would replace

sort --random-sort /root/ips.csv | head -n 1

by

entire_pipe 'sort --random-sort /root/ips.csv' 'head -n 1'

This is supposed to work even if the left side of pipe starts descendant processes; they will be all interrupted.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.