Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The problem has been localized to the reading of a variable in the BEGIN block of an awk statement at the end of the pipe line. During some executions, the variable is correctly read in the BEGIN block and during other executions, the operation fails. Supposing this aberrant behavior can be reproduced by others (and is not a consequence of some problem with my system), can its inconsistency be explained?

Take as input the following file called tmp:

cat > tmp <<EOF
a   a
b   *
aa  a
aaa a
aa  a
a   a
c   *
aaa a
aaaa    a
d   *
aaa a
a   a
aaaaa   a
e   *
aaaa    a
aaa a
f   *
aa  a
a   a
g   *
EOF

On my system, the pipe line

 awk '{if($2!~/\*/) print $1}' tmp | tee >(wc -l | awk '{print $1}' > n.txt) | sort | uniq -c | sort -k 1,1nr | awk 'BEGIN{getline n < "n.txt"}{print $1 "\t" $1/n*100 "\t" $2}'

will either produce the correct output:

4   28.5714 a
4   28.5714 aaa
3   21.4286 aa
2   14.2857 aaaa
1   7.14286 aaaaa

or the error message:

awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted

How can a command possibly give different output when run twice in succession when no random number generation is involved and no change to the environment is made in the interim?

To demonstrate how absurd the behavior is, consider the output generated by executing the above pipe line ten times consecutively in a loop:

for x in {1..10}; do echo "Iteration ${x}"; awk '{if($2!~/\*/) print $1}' tmp | tee >(wc -l | awk '{print $1}' > n.txt) | sort | uniq -c | sort -k 1,1nr | awk 'BEGIN{getline n < "n.txt"}{print $1 "\t" $1/n*100 "\t" $2}'; done
Iteration 1
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
Iteration 2
4   28.5714 a
4   28.5714 aaa
3   21.4286 aa
2   14.2857 aaaa
1   7.14286 aaaaa
Iteration 3
4   28.5714 a
4   28.5714 aaa
3   21.4286 aa
2   14.2857 aaaa
1   7.14286 aaaaa
Iteration 4
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
Iteration 5
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
Iteration 6
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
Iteration 7
4   28.5714 a
4   28.5714 aaa
3   21.4286 aa
2   14.2857 aaaa
1   7.14286 aaaaa
Iteration 8
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted
Iteration 9
4   28.5714 a
4   28.5714 aaa
3   21.4286 aa
2   14.2857 aaaa
1   7.14286 aaaaa
Iteration 10
awk: cmd. line:1: (FILENAME=- FNR=1) fatal: division by zero attempted

Note: I have also tried closing the file (awk close) after reading the variable, in case the problem relates to the file being left open. However, the inconsistent output remains.

share|improve this question
    
not sure yet on how to fix this, but I know the error is b/c this portion: $1/n*100, specifically by n. If for example, you remove that n your script runs fine. This should be related to n value read from n.txt. –  Simply_Me Aug 11 at 2:07
    
You're trying to count the occurrences of lines in a file? Like 28.5714 a occurs 4 times? I think you'd have a lot more luck with uniq or grep -c. My bad - I see that pipeline is a lot longer than I at first realized - and I mostly have the same question that ^ previous guy did - n? It some kind of formulae related to percentage like 4 times is close to 27% - but there's something else. 2% else. Anyway, I'm sure it can be much simplified and I urge you to try. sort - for instance - is never a back-pipe friendly process. –  mikeserv Aug 11 at 8:30

3 Answers 3

up vote 13 down vote accepted

Your redirections have a race condition. This:

>(wc -l | awk '{print $1}' > n.txt)

runs in parallel with:

awk 'BEGIN{getline n < "n.txt"}...'

later in the pipeline. Sometimes, n.txt is still empty when the awk program starts running.

This is (obliquely) documented in the Bash Reference Manual. In a pipeline:

The output of each command in the pipeline is connected via a pipe to the input of the next command. That is, each command reads the previous command’s output. This connection is performed before any redirections specified by the command.

and then:

Each command in a pipeline is executed in its own subshell

(emphasis added). All the processes in the pipeline are started, with their input and output connected together, without waiting for any of the earlier programs to finish or even start doing anything. Before that, process substitution with >(...) is:

performed simultaneously with parameter and variable expansion, command substitution, and arithmetic expansion.

What that means is that the subprocess running the wc -l | awk ... command starts early on, and the redirection empties n.txt just before that, but the awk process that causes the error is started shortly after. Both of those commands execute in parallel - you'll have several processes going at once here.

The error occurs when awk runs its BEGIN block before the wc command's output has been written into n.txt. In that case, the n variable is empty, and so is zero when used as a number. If the BEGIN runs after the file is filled in, everything works.

When that happens depends on the operating system scheduler, and which process gets a slot first, which is essentially random from the user perspective. If the final awk gets to run early, or the wc pipeline gets scheduled a little later, the file will still be empty when awk starts doing its work and the whole thing will break. In all likelihood the processes will run on different cores actually simultaneously, and it's down to which one gets to the point of contention first. The effect you'll get is probably of the command working more often than not, but sometimes failing with the error you post.


In general, pipelines are only safe in so far as they're just pipelines - standard output into standard input is fine, but because the processes execute in parallel it's not reliable to rely on the sequencing of any other communication channels, like files, or of any part of any one process executing before or after any part of another unless they're locked together by reading standard input.

The workaround here is probably to do all your file writing in advance of needing them: at the end of a line, it's guaranteed that an entire pipeline and all of its redirections have completed before the next command runs. This command will never be reliable, but if you really do need it to work in this sort of a structure you can insert a delay (sleep) or loop until n.txt is non-empty before running the final awk command to increase the chances of things working how you want.

share|improve this answer
    
Thanks for a truly exceptional explanation. I was laboring under a flawed understanding of pipelines, which I erroneously perceived to operate in a completely serial fashion. It's true that the tee statement produces parallel processes, so even though I wrote the process for variable definition first, there is no reason to expect that it will begin or finish first. Also, I had no idea that the pipes were constructed at the outset, and instead naïvely believed that each pipe was constructed after completion of the preceding process. There is no fundamental reason that I need to employ the ... –  user001 Aug 11 at 2:30
    
strategy in the example. I was merely trying to condense the workflow into a "one-liner" more for the concision of such a formulation than anything else. Initially, I wanted to avoid writing temporary files as well and tried to pipe the output of the >(...) process substitution to a read statement to subsequently pass the variable as an argument to the later awk script, only to discover that read cannot be used in this process because a child process, the state of the >(...) construct, has not the ability to alter the environment of its parent process. –  user001 Aug 11 at 2:35

pipe expression in process substitution causes a race condition in bash and ksh, zsh doesn't.

The main problem here is that zsh waits, bash doesn't.

You can see more details here.

A quick fixed, adding sleep 1 in your awk to make n.txt always available:

awk 'BEGIN{system("sleep 1");getline n < "n.txt"}{print $1 "\t" $1/n*100 "\t" $2}'
share|improve this answer
    
Thanks for linking to an interesting discussion. I am inclined to favor the zsh model of synchronizing the processes. It's interesting that the first process after the tee will finish first if a sleep is imposed, but finish either first or second (pseudo-randomly) if not. –  user001 Aug 11 at 2:51

The race condition is already identified. But if you'd like an easier solution, you don't need a separate wc to count the records, awk can do it:

awk '{if($2!~/\*/){print $1;++n}END{print n >"n.txt"}' tmp | sort | uniq -c ...

Beyond that, awk can count like sort|uniq -c as long as the values fit in memory, and also do the x/n calculation, but may output in "random" order; also using match/action is tidier:

awk '$2!~/\*/{++k[$1];++n} END{for(i in k){print k[i]"\t"k[i]/n*100"\t"i}}' tmp | sort -k1nr

Or in recent GNU awk you can set PROCINFO["sorted_in"]="@ind_num_desc" so the for uses the correct order and you don't need the sort.

share|improve this answer
    
Thanks, that's a good idea. I'll use that. Particularly when a process is quite compute- or IO-intensive (my actual data files are gigabytes), obviating the need for process re-execution (saving a result from the first execution in a variable or temporary file to be read and used during the second execution) is desirable. Thanks for mentioning the preferred match/action formulation as well. I'm aware of it, but rarely make use of it, as resorting to the if statement has become a pernicious habit of mine. –  user001 Aug 11 at 16:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.