Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm trying to read a text file and do something with each line, using a bash script.

So, I have a list that looks like this:

server1
server2
server3
server4

I thought I could loop over this using a while loop, like so:

while read server; do
  ssh $server "uname -a"
done < /home/kenny/list_of_servers.txt

The while loop stops after 1 run, thus only running uname -a on server1

However, with a for loop using cat it works fine:

for server in $(cat /home/kenny/list_of_servers.txt) ; do
  ssh $server "uname -a"
done

Even more baffling to me is that this also works:

while read server; do
  echo $server
done < /home/kenny/list_of_servers.txt

Why does my first example stop after the first iteration?

share|improve this question
add comment

3 Answers

up vote 13 down vote accepted

The for loop is fine here. But note that this is because the file contains machine names, which do not contain any whitespace characters or globbing characters. for x in $(cat file); do … does not work to iterate over the lines of file in general, because the shell first splits the output from the command cat file anywhere there is whitespace, and then treats each word as a glob pattern so \[?* are further expanded. You can make for x in $(cat file) safe if you work on it:

set -f
IFS='
'
for x in $(cat file); do …

Related reading: Looping through files with spaces in the names?; How can I read line by line from a variable in bash?; Why is while IFS= read used so often, instead of IFS=; while read..? Note that when using while read, the safe syntax to read lines is while IFS= read -r line; do ….

Now let's turn to what goes wrong with your while read attempt. The redirection from the server list file applies to the whole loop. So when ssh runs, its standard input comes from that file. The ssh client can't know when the remote application might want to read from its standard input. So as soon as the ssh client notices some input, it sends that input to the remote side. The ssh server there is then ready to feed that input to the remote command, should it want it. In your case, the remote command never reads any input, so the data ends up discarded, but the client side doesn't know anything about that. Your attempt with echo worked because echo never reads any input, it leaves its standard input alone.

There are a few ways you can avoid this. You can tell ssh not to read from standard input, with the -n option.

while read server; do
  ssh -n $server "uname -a"
done < /home/kenny/list_of_servers.txt

The -n option in fact tells ssh to redirect its input from /dev/null. You can do that at the shell level, and it'll work for any command.

while read server; do
  ssh $server "uname -a" </dev/null
done < /home/kenny/list_of_servers.txt

A tempting method to avoid ssh's input coming from the file is to put the redirection on the read command: while read server </home/kenny/list_of_servers.txt; do …. This will not work, because it causes the file to be opened again each time the read command is executed (so it would read the first line of the file over and over). The redirection needs to be on the whole while loop so that the file is opened once for the duration of the loop.

The general solution is to provide the input to the loop on a file descriptor other than standard input. The shell has constructs to ferry input and output from one descriptor number to another. Here, we open the file on file descriptor 3, and redirect the read command's standard input from file descriptor 3. The ssh client ignores open non-standard descriptors, so all is well.

while read server <&3; do
  ssh $server "uname -a"
done 3</home/kenny/list_of_servers.txt

In bash, the read command has a specific option to read from a different file descriptor, so you can write read -u3 server.

Related reading: File descriptors & shell scripting; When would you use an additional file descriptor?

share|improve this answer
    
Man, this stackexchange sure is different from serverfault. People here really go out of their way to not only answer, but also educate. Thanks a lot. –  Kenny Rasschaert Nov 10 '11 at 6:27
add comment

You should use while, not for. The way to avoid commands swallowing standard input in such a loop is simply to use another file descriptor:

while read -u 9 server; do
  ssh $server "uname -a"
done 9< /home/kenny/list_of_servers.txt
share|improve this answer
    
Interesting, I've never used file descriptors before. What does the -u flag do? I'm not sure in what man page I'm supposed to look it up. –  Kenny Rasschaert Nov 9 '11 at 13:59
    
The read command is part of the shell bash, so it's in the man page of bash in section "SHELL BUILTIN COMMANDS" in the read paragraph. You will find "-u fd Read input from file descriptor fd." in this paragraph –  f4m8 Nov 9 '11 at 14:24
6  
Alternatively help read - help is the command to get the equivalent of man pages for shell builtins. –  l0b0 Nov 9 '11 at 14:33
add comment

In your first code ssh will “steal” the STDIN from the while. Add -n option to ssh to avoid it. From man ssh:

-n     Redirects stdin from /dev/null (actually, prevents reading from stdin).
share|improve this answer
    
Ok, got any idea why this doesn't happen with the for loop? –  Kenny Rasschaert Nov 9 '11 at 13:21
6  
Because the for loop receives the data as parameter, not as input. –  manatwork Nov 9 '11 at 13:25
1  
You sir, are a gentleman and a scholar. –  Kenny Rasschaert Nov 9 '11 at 13:32
    
+1 Nice, and simpler than my solution. –  l0b0 Mar 8 '12 at 12:11
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.