0

I'm playing around with the accepted answer from this thread: Bash script that reads filenames from a pipe or from command line args?

When I use the below script, efetch (ftp://ftp.ncbi.nlm.nih.gov/entrez/entrezdirect/edirect.zip) accepts an id (argument, for example, 941241313) but not stdin.

if [ $# -gt 0 ] ;then
    for name in "$@"; do
        efetch -db nucleotide -id $name -format gpc > $name.xml;
    done
  else
    IFS=$'\n' read -d '' -r -a filenames
    while read name; do
        efetch -db nucleotide -id $name -format gpc > $name.xml;
    done < "${filenames[@]}"
  fi

When I modify it to the below version, efetch accepts stdin instead of an id

if [ $# -gt 0 ] ;then
    while read name; do
        efetch -db nucleotide -id $name -format gpc > $name.xml;
    done < "$@"
  else
    IFS=$'\n' read -d '' -r -a filenames
    while read name; do
        efetch -db nucleotide -id $name -format gpc > $name.xml;
    done < "${filenames[@]}"
  fi

What's wrong?

1
  • 1
    You can't read from multiple files by feeding an array expansion to the < operator; that's just a syntax error.
    – chepner
    Nov 12 '15 at 14:44
1

The change is to this chunk

while read name; do
    efetch -db nucleotide -id $name -format gpc > $name.xml;
done < "$@"

which makes the efetch in the loop run with its standard input redirected to the file given by the arguments. So that makes two changes to the way efetch is used:

  • its standard input is no longer the default (terminal)
  • its list of parameters is no longer literally the command-line parameters for the script, but indirectly, from a file.

If efetch detects that its input is not a terminal, it could very well reopen the terminal directly (perhaps that is what you are referring to as "efetch accepts stdin instead of an id"). Alternatively, if efetch is reading its stdin, it could read something unexpected (in a quick test, that seems to be the script itself).

@chepner pointed out that the shell (bash in this case) does not spawn a subprocess for the loop. I had in mind a different case which does. Consider these two scripts:

#!/bin/bash 
LAST=...
while read name
do
    /bin/echo "** $name"
    LAST="$name"
done < "$@"
echo "...$LAST"

and

#!/bin/bash
LAST=...
cat "$@" | while read name
do
    /bin/echo "** $name"
    LAST="$name"
done
echo "...$LAST"

The latter (pipe) will echo "......" at the end, while the former (redirection) echoes the last variable assigned to LAST within the loop. The form using a pipe is sometimes commented on as requiring a subprocess to account for the reason why variable assignments are not propagated out of the loop.

Interestingly enough, there are differences between shells for the latter (a pipe) regarding the number of processes used. Testing with (Debian/testing) bash, dash (/bin/sh), zsh and ksh93, using strace -fo to capture system calls and process ids:

#!/bin/sh
for sh in bash dash zsh ksh93
do
    echo "++ $sh"
    strace -fo $sh.log ./do-$sh ./once
    LC=$(sed -e 's/ .*//' $sh.log |sort -u |wc -l)
    WC=$(wc -l $sh.log)
    echo "-- $LC / $WC"
done

The script shows the number of processes and the number of system calls for each shell. (The file once contains two lines: "first" and "second", to eliminate one testing boundary).

I see that zsh and ksh93 use one process fewer than bash and dash:

$ ./testit
++ bash
** first
** second
......
-- 5 / 401 bash.log
++ dash
** first
** second
......
-- 5 / 222 dash.log
++ zsh
** first
** second
...second
-- 4 / 568 zsh.log
++ ksh93
** first
** second
...second
-- 4 / 336 ksh93.log

Running the pipe takes 1 or 2 more processes than using a here-document for this example.

2
  • There's no (additional) subprocess involved with redirection, but the point about efetch detecting if its standard input is a terminal or not stands.
    – chepner
    Nov 12 '15 at 14:42
  • I see (will amend). Nov 13 '15 at 0:34
0

My bad, as pointed out by @chepner, this block of code is not working at all. For some reasons, my script was broken. It was done using nano in a GNU screen session (in a server). efetch can still be ran after the lines have been commented out!

IFS=$'\n' read -d '' -r -a filenames
while read name; do
    efetch -db nucleotide -id $name -format gpc > $name.xml;
done < "${filenames[@]}"

The below script is ugly but it works.

#!/bin/bash
if [ $# -eq 0 ]; then
    echo "Please provide either a GI or a list of GI (one per line) in a file!"
elif [ -r "$@" ]; then
    while read name; do
        efetch -db nucleotide -id $name -format gpc > $name.xml;
    done < "$@"
else
    efetch -db nucleotide -id "$@" -format gpc > "$@".xml;
fi

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.