0

I'm doing something along these lines:

declare -Ft handle_format_output &>/dev/null && exit 1   # test if this function name is already present in this scope
handle_format_output() {
    case $1 in
    domain) ./scripts/idn_to_punycode.pl >>"${2}_${1}.txt" ;;
    ipv4)
        # read doesn't bother w/ pipe input as its undefined: guessing it's leading to pipe breaks
        # therefore open stdin as a separate file and read from it to close the previous pipe
        while IFS= read -r line <&3; do
            case $line in
            */*) printf "%s\n" "$line" >>"${2}_${1}_cidr.txt" ;; # cidr block
            *-*) ipcalc "$line" >>"${2}_${1}_cidr.txt" ;;        # deaggregate ip range
            *) printf "%s\n" "$line" >>"${2}_${1}.txt" ;;        # ip address
            esac
        done 3< /dev/stdin
        ;;
    ipv6) cat -s >>"${2}_${1}.txt" ;;
    esac
}

...  |
  mawk '!seen[$0]++' | # filter duplicates and blank lines
  handle_format_output "$format" "$color"

Where the input is linear text that's either a web domain, IPv4 address, IPv4 CIDR block, or IPv6 address. The format is either "domain," "ipv4," or "ipv6." The color is either "white" or "black."

This error keeps being thrown no matter what I'm trying:

mawk: write failure (Broken pipe)
mawk: close failed on file /dev/stdout (Broken pipe)
Error: Process completed with exit code 2.

What am I doing wrong?

4
  • What are the values of $format and $color? (please double check these and mention them in the question).
    – Kusalananda
    Jul 3, 2021 at 15:36
  • @Kusalananda I've added those details in the question
    – T145
    Jul 3, 2021 at 15:39
  • Could you please confirm that $format is actually exactly the string ipv4 and nothing else (no trailing spaces or newline etc.)? You seem to think there's something wrong with your read call, so the first step is to make sure that the code actually gets there. If $format is a string that is not one of the three that you test for, the code will fall through that case statement and you'll get the behavior that you describe. You may therefore want to add a *) case that just prints error! or something, for debugging.
    – Kusalananda
    Jul 3, 2021 at 16:36
  • @Kusalananda If anything were different or more relevant it would've been documented.
    – T145
    Jul 3, 2021 at 17:23

1 Answer 1

0

I figured it out! Some of the format data fields were "ivp4" instead of "ipv4." I'm now wondering why there have been successful builds this long then, but after fixing them regular piping now works.

7
  • There is literally no difference between the code in your answer and the code in you question. In both instances, your function reads the data from its standard input. The input is not "completely processed first" in your answer's code. It's redirected from a named pipe, which is effectively the same as piping it into the function. You obviously changed something else in the code or in the input data that made it work.
    – Kusalananda
    Jul 3, 2021 at 17:18
  • @Kusalananda I've changed nothing other than this line, and no error is thrown. From what I understand, this version waits until all the text is processed from the other commands, and passes the final output into the function. Before, there was still some processing going on, which is why the pipe broke when processing the output.
    – T145
    Jul 3, 2021 at 17:22
  • I don't really buy that as a reason unless there were some timeout while reading the data. Your code shows nothing that should be sensitive to timing out or reading lines with slight delays.
    – Kusalananda
    Jul 3, 2021 at 19:57
  • @Kusalananda Here's literally all I've changed if you want the full context: github.com/T145/the-blacklist/commit/… After that change everything built perfectly, so if there's any way to do things better I'd love to learn it.
    – T145
    Jul 3, 2021 at 20:13
  • Try comparing { while true; do echo hello; sleep 1; done; } | while read; do echo "$REPLY"; done and while read; do echo "$REPLY"; done < <( while true; do echo hello; sleep 1; done ) and note how the second of these does not finish processing before sending the data to the read (the output is the same in both example, the string hello is repeatedly printed with a delay). It is not clear that you are providing the new code with the same input as the old code.
    – Kusalananda
    Jul 3, 2021 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.