11

We have multiple deployment of an application on servers such as app00, app01 and so on. I need to copy a single log file from all these servers onto my local mac so I can perform some grepping and cutting.

I used csshX for viewing this file but I cannot find an equivalent for scp. I basically want two things:

  1. Ability to connect to n numbers of such servers and copy the file
  2. Avoid naming conflicts locally perhaps by prefixing the log file with the server hostname

How do I do this?

2
  • 1
    I would probably write a script for it myself. Is that not an option?
    – Bernhard
    May 20, 2014 at 11:32
  • I am doing that right now Bernhard. :) I was just hoping there was something like csshX for this also.
    – Gaurav
    May 20, 2014 at 11:37

5 Answers 5

18

This is trivial to do with a little script. For example:

for server in app0 app1 app4 app5 appN; do
    scp user@$server:/path/to/log/file /local/path/to/"$server"_file
done

The above will copy the file from each of the servers sequentially and name it SERVERNAME_file. So, the file from app0 will be app0_file etc. You can obviously change the names to whatever you would like.

2
  • 3
    Put & at the end of the scp command and a wait at the end, and you have concurrency at no extra cost.
    – l0b0
    May 20, 2014 at 17:21
  • 3
    @l0b0 I don't know if that's desired. I like this approach because I can know if and which files failed to copy. Good tip though, thanks.
    – terdon
    May 20, 2014 at 17:25
13

Use GNU parallel:

parallel -j0 scp {}:/remote_path file_from_{} ::: host1 host2 host3 # and so on

Unlike solutions that use a for, this will run all the downloads in parallel

5
remote_path="/path/to/file"
local_target_dir="/path/to/dir"
hosts=(app00 app01)
for host in "${hosts[@]}"; do
    scp "$host":"$remote_path" "$local_target_dir"/filename."$host"
done
1
  • Thanks a lot, I am new to bash programming and wrote something like that, just a lot dirtier. :)
    – Gaurav
    May 20, 2014 at 11:37
0

If you are able to use python, there is an interesting module that looks to simplify machine administration-ish tasks called fabric: http://docs.fabfile.org/en/latest/tutorial.html

I have wanted to use it but have not got around too it

0

This worked for me

#!/bin/bash
#Expect script
/usr/bin/expect -<<EOD 

set SERVERS {1 2 3 .. N}

foreach SERVER \$SERVERS {
    spawn scp user@\$SERVER:remote local/"\$SERVER"RESWeb.log
    expect {
      -re ".*es.*o.*" {
        exp_send "yes\r"
        exp_continue
      }
      -re ".*sword.*" {
        exp_send "pswrd\r"
      }
    }
    expect eof 
}
EOD

echo "completed"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .