We have multiple deployment of an application on servers such as app00, app01 and so on. I need to copy a single log file from all these servers onto my local mac so I can perform some grepping and cutting.

I used csshX for viewing this file but I cannot find an equivalent for scp. I basically want two things:

  1. Ability to connect to n numbers of such servers and copy the file
  2. Avoid naming conflicts locally perhaps by prefixing the log file with the server hostname

How do I do this?

  • 1
    I would probably write a script for it myself. Is that not an option? – Bernhard May 20 '14 at 11:32
  • I am doing that right now Bernhard. :) I was just hoping there was something like csshX for this also. – Gaurav May 20 '14 at 11:37

This is trivial to do with a little script. For example:

for server in app0 app1 app4 app5 appN; do
    scp user@$server:/path/to/log/file /local/path/to/"$server"_file

The above will copy the file from each of the servers sequentially and name it SERVERNAME_file. So, the file from app0 will be app0_file etc. You can obviously change the names to whatever you would like.

|improve this answer|||||
  • 3
    Put & at the end of the scp command and a wait at the end, and you have concurrency at no extra cost. – l0b0 May 20 '14 at 17:21
  • 3
    @l0b0 I don't know if that's desired. I like this approach because I can know if and which files failed to copy. Good tip though, thanks. – terdon May 20 '14 at 17:25

Use GNU parallel:

parallel -j0 scp {}:/remote_path file_from_{} ::: host1 host2 host3 # and so on

Unlike solutions that use a for, this will run all the downloads in parallel

|improve this answer|||||
hosts=(app00 app01)
for host in "${hosts[@]}"; do
    scp "$host":"$remote_path" "$local_target_dir"/filename."$host"
|improve this answer|||||
  • Thanks a lot, I am new to bash programming and wrote something like that, just a lot dirtier. :) – Gaurav May 20 '14 at 11:37

If you are able to use python, there is an interesting module that looks to simplify machine administration-ish tasks called fabric: http://docs.fabfile.org/en/latest/tutorial.html

I have wanted to use it but have not got around too it

|improve this answer|||||

This worked for me

#Expect script
/usr/bin/expect -<<EOD 

set SERVERS {1 2 3 .. N}

foreach SERVER \$SERVERS {
    spawn scp user@\$SERVER:remote local/"\$SERVER"RESWeb.log
    expect {
      -re ".*es.*o.*" {
        exp_send "yes\r"
      -re ".*sword.*" {
        exp_send "pswrd\r"
    expect eof 

echo "completed"
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.