Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am running my shell script on machineA which copies the files from machineB and machineC to machineA.

If the file is not there in machineB, then it should be there in machineC for sure. So I will try to copy from machineB first, if it is not there in machineB then I will go to machineC to copy the same files.

In machineB and machineC there will be a folder like this YYYYMMDD inside this folder -

/data/pe_t1_snapshot

So whatever date is the latest date in this format YYYYMMDD inside the above folder - I will pick that folder as the full path from where I need to start copying the files -

so suppose if this is the latest date folder 20140317 inside /data/pe_t1_snapshot then this will be the full path for me -

/data/pe_t1_snapshot/20140317

from where I need to start copying the files in machineB and machineC. I need to copy around 400 files in machineA from machineB and machineC and each file size is 1.5 GB.

Currently I have my below shell script which works fine as I am using scp over rsync but somehow it takes 5 hours to copy the 400 files in machineA which is too long for me I guess. :(

Below is my shell script -

#!/bin/bash

readonly PRIMARY=/export/home/david/dist/primary
readonly SECONDARY=/export/home/david/dist/secondary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot
PRIMARY_PARTITION=(0 3 5 7 9)
SECONDARY_PARTITION=(1 2 4 6 8)

dir1=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[0]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[1]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

if [ "$dir1" = "$dir2" ]
then
    # delete all the files first
    rm -rf $PRIMARY/*
    # below for-loop copies one file at a time in PRIMARY folder
    for el in "${PRIMARY_PARTITION[@]}"
    do
        scp david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/. || scp david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$el"_200003_5.data $PRIMARY/.
    done
    # delete all the files first
    rm -rf $SECONDARY/*
    # below for-loop copies one file at a time in SECONDARY folder
    for sl in "${SECONDARY_PARTITION[@]}"
    do
        scp david@${FILERS_LOCATION[0]}:$dir1/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/. || scp david@${FILERS_LOCATION[1]}:$dir2/t1_weekly_1680_"$sl"_200003_5.data $SECONDARY/.
    done
fi

I am copying PRIMARY_PARTITION files in PRIMARY folder and SECONDARY_PARTITION files in SECONDARY folder in machineA.

Now my question is - how would I use rsync here instead of scp(ing) the files? As I have read that rsync is much faster than scp(ing) the files. I would like to have same logic as I have in my shell script in rsync. I have never worked with rsync before so having some issue.

Can anyone provide an example?

Given my use case does rsync will be faster as compare to scp? And if not what other options I can try out to speed up the file transfer?

UPDATE:-

To clarify terdon question -

In the question I am only showing 10 files just for example -

PRIMARY_PARTITION=(0 3 5 7 9)
SECONDARY_PARTITION=(1 2 4 6 8)

In general in PRIMARY_PARTITION array, I will around 150 file numbers and then in SECONDARY_PARTITION I will have another 200 files numbers.

Now what I need to do is whatever file number I have in PRIMARY_PARTITION, I need to go and find out those files in the machineB directory, if the files is already there, then copy it to PRIMARY folder in machineA but if the file is not there in machineB then it should be there in machineC so go and copy the files from machineC now and put it in PRIMARY folder in machineA.

Similarly I need to do same thing for SECONDARY_PARTITION, I will go and find out those files in machineB directory, if it is there copy it in machineA secondary directory but if it is not there in machineB, then it should be there in machineC so copy it from machineC and put it in machineA secondary directory.

So all the file numbers we have is in - PRIMARY_PARTITION and SECONDARY_PARTITION.

In general I will have PRIMARY_PARTITION and SECONDARY_PARTITION like this -

PRIMARY_PARTITION=(0 548 272 4 544 276 8 556 280 12 552 284 16 256 564 20 260 560 24 264 572 28 268 568 516 304 32 512 308 36 524 312 40 520 316 44 288 532 48 292 528 52 296 540 56 300 536 60 68 608 340 64 336 76 348 72 344 84 324 80 320 92 332 88 328 576 372 100 580 368 96 584 380 108 588 376 104 356 592 116 352 596 112 364 600 124 360 604 120 136 408 140 412 128 400 132 404 152 392 156 396 144 384 148 388 440 168 444 172 432 160 436 164 424 184 428 188 416 176 420 180 204 476 200 472 196 468 192 464 220 460 216 456 212 452 208 448 508 236 504 232 500 228 496 224 492 252 488 248 484 244 480 240)

SECONDARY_PARTITION=(1101 1374 1641 1371 1647 1098 1635 1365 1095 1638 1089 1362 1659 1359 1119 1113 1662 1353 1350 1650 1110 1347 1653 1107 1134 1407 1611 1401 1131 1614 1602 1125 1398 1122 1605 1395 1389 1149 1626 1629 1146 1386 1617 1143 1383 1377 1623 1137 1305 1581 1578 1311 1299 1575 1302 1569 1599 1290 1593 1293 1590 1281 1587 1287 1551 1338 1341 1545 1071 1329 1542 1335 1539 1083 1566 1323 1086 1563 1326 1557 1074 1314 1317 1077 1554 1221 1494 1491 1218 1503 1230 1227 1497 1479 1239 1233 1473 1245 1485 1482 1242 1254 1527 1251 1521 1263 1533 1530 1257 1509 1269 1266 1506 1278 1518 1275 1515 1155 1425 1431 1158 1434 1161 1167 1437 1410 1170 1173 1413 1419 1179 1422 1182 1671 1458 1185 1665 1191 1461 1677 1194 1467 1470 1197 1674 1203 1443 1206 1446 1449 1209 1215 1455)

Another Update:-

After removing 2>/dev/null, I ran the script again but I got the below error -

ssh: Could not resolve hostname machineB : Name or service not known rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9] ssh: Could not resolve hostname machineC : Name or service not known rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9] ssh: Could not resolve hostname machineB : Name or service not known rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9] ssh: Could not resolve hostname machineC : Name or service not known rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(605) [Receiver=3.0.9]

Any thoughts what is going on? I have replaced machineB and machineC with actual names before running the shell script and my system is -

root@machineA:/home/david# uname -a
Linux machineA 3.2.0-24-generic #37-Ubuntu SMP Wed Apr 25 08:43:22 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Below is my shell script which I am running -

#!/usr/bin/env bash

readonly PRIMARY=/export/home/david/dist/primary
readonly SECONDARY=/export/home/david/dist/secondary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot
PRIMARY_PARTITION=(0 3 5 7 9)
SECONDARY_PARTITION=(1 2 4 6 8)

dir1=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[0]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[1]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

## Build your list of filenames before the loop. 
for n in "${PRIMARY_PARTITION[@]}"
do
    primary_files="$primary_files :$dir1"/t1_weekly_1680_"$n"_200003_5.data
done

## Repeat for $SECONDARY_PARTITION
for n in "${SECONDARY_PARTITION[@]}"
do
    secondary_files="$secondary_files :$dir2"/t1_weekly_1680_"$n"_200003_5.data
done

echo $primary_files
echo $secondary_files


if [ "$dir1" = "$dir2" ]
then
    find "$PRIMARY" -mindepth 1 -delete
    find "$SECONDARY" -mindepth 1 -delete

    rsync -avz david@${FILERS_LOCATION[0]}"${primary_files}" $PRIMARY/
    rsync -avz david@${FILERS_LOCATION[1]}"${primary_files}" $PRIMARY/

    ## Do the same for $secondary_partition files
    rsync -avz david@${FILERS_LOCATION[0]}"${secondary_files}" $SECONDARY/
    rsync -avz david@${FILERS_LOCATION[1]}"${secondary_files}" $SECONDARY/
fi

I am suspecting may be rsync syntax is not right. Because if I run single command like this, then it worked fine -

rsync -avz david@machineB":/data/pe_t1_snapshot/20140317/t1_weekly_1680_0_200003_5.data" /export/home/david/dist/primary

Another Minor Update:-

If I am running like this -

root@machineA:/export/home/david# rsync -avz david@machineB':/data/pe_t1_snapshot/20140317/t1_weekly_1680_0_200003_5.data :/data/pe_t1_snapshot/20140317/t1_weekly_1680_1_200003_5.data' /data01/primary
receiving incremental file list
rsync: change_dir "/home/david/:/data/pe_t1_snapshot/20140317" failed: No such file or directory (2)
t1_weekly_1680_0_200003_5.data

sent 30 bytes  received 504982813 bytes  6196108.50 bytes/sec
total size is 1761988281  speedup is 3.49
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1536) [generator=3.0.9]

Above command should copy the files to /data01/primary directory but it is copying only one file and it doesn't copies second file.

But this works fine and the one file get copied -

root@machineA:/export/home/david# rsync -avz david@machineB':/data/pe_t1_snapshot/20140317/t1_weekly_1680_0_200003_5.data' /data01/primary
receiving incremental file list
t1_weekly_1680_0_200003_5.data

sent 30 bytes  received 504982698 bytes  6351984.00 bytes/sec
total size is 1761988281  speedup is 3.49
share|improve this question
    
Once you have identified the folder you want to copy from, are you going to want all the files from it or will there be files you don't want to copy? Also, you really don't want the same logic, your main problem here is that the logic is wrong, you're opening one or even two connections per file transferred, that's why it's taking so long. How can this script copy 400 files? What you show will only ever copy 10 files: t1_weekly_1680_0_200003_5.data to t1_weekly_1680_9_200003_5.data. Where are the other 390 coming from? –  terdon Mar 20 at 22:54
    
@terdon: I have updated the question with more details for you. The 10 file was just an example to explain what I am doing, it will have more files. I need to copy files basis on PRIMARY_PARTITION and SECONDARY_PARTITION as these two contains file number which I am suppose to copy. –  Webby Mar 21 at 2:07
    
You don't need to ping me twice, if you leave one message with my @name in it, I'll see it. Will there be files in the target directory you don't want to copy? –  terdon Mar 21 at 2:12
    
@terdon: Sorry about that. Yes there will be some files which I don't want to copy.. In this folder /data/pe_t1_snapshot/20140317 there will be lot of files but I want to copy only those files which are there in PRIMARY_PARTITION and SECONDARY_PARTITION variable. –  Webby Mar 21 at 2:15
    
Did you also try a double command? Like this: rsync -avz david@machineB:/data/pe_t1_snapshot/20140317/t1_weekly_1680_0_200003_5.data :/data/pe_t1_snapshot/20140317/t1_weekly_1680_1_200003_5.data/export/home/david/‌​dist/primary? That syntax works as long as you have a relatively new rsync version, older ones need rsync -av host:'dir1/file1 dir2/file2' /dest. –  terdon Mar 21 at 13:56

3 Answers 3

up vote 1 down vote accepted

The main problem with your script is that you're opening a separate scp connection for each file, that adds a lot of needless overhead. You could try something like this instead:

#!/usr/bin/env bash

readonly PRIMARY=/export/home/david/dist/primary
readonly SECONDARY=/export/home/david/dist/secondary
readonly FILERS_LOCATION=(machineB machineC)
readonly MEMORY_MAPPED_LOCATION=/data/pe_t1_snapshot

PRIMARY_PARTITION=(0 548 272 4 544 276 8 556 280 12 552 284 16 256 564 20 260 560 24 264 572)
SECONDARY_PARTITION=(1101 1374 1641 1371 1647 1098 1635 1365 1095 1638 1089 1362 1659 1359)

dir1=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[0]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${FILERS_LOCATION[1]} ls -dt1 "$MEMORY_MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

## Build your list of filenames before the loop. 
for n in "${PRIMARY_PARTITION[@]}"
do
    primary_files="$primary_files :$dir1"/t1_weekly_1680_"$n"_200003_5.data
done

## Repeat for $SECONDARY_PARTITION
for n in "${SECONDARY_PARTITION[@]}"
do
    secondary_files="$secondary_files :$dir2"/t1_weekly_1680_"$n"_200003_5.data
done

if [ "$dir1" = "$dir2" ]
then
    ## I am using find largely because the * 
    ## in rm -rf "$PRIMARY"/* screws up the syntax 
    ## highlighting on the site and it is a good habit to
    ## get into anyway. Feel free to use rm -rf in your script.
    find "$PRIMARY" -mindepth 1 -delete
    find "$SECONDARY" -mindepth 1 -delete

    ## rsync can be run with this format:
    ##   rsync user@dest:/target/path1 :/target/path2 :/target/pathN /dest/path
    #
    ## which is why I added the : in the loop above. So, these commands will 
    ## open only 2 conections per file list. First you will try to copy all $primary_partition
    ## files from machineA, then all $primary_partition files from machineB. 
    ## rsync will complain about files not found (which is why I'm redirecting standard
    ## error to /dev/null) but will continue. You then repeat the process for machineC.
    rsync -avz david@${FILERS_LOCATION[0]}"${primary_files}" $PRIMARY/ 2>/dev/null
    rsync -avz david@${FILERS_LOCATION[1]}"${primary_files}" $PRIMARY/ 2>/dev/null

    ## Do the same for $secondary_partition files
    rsync -avz david@${FILERS_LOCATION[0]}"${secondary_files}" $SECONDARY/ 2>/dev/null
    rsync -avz david@${FILERS_LOCATION[1]}"${secondary_files}" $SECONDARY/ 2>/dev/null
fi
share|improve this answer
    
Thanks a lot for the help.. Quick question - what does this line find "$PRIMARY" -delete do? It will delete everything inside this folder /export/home/david/dist/primary or it will delete this folder /export/home/david/dist/primary and everything inside that? because I need to delete everything inside this folder /export/home/david/dist/primary but don't delete the primary folder. Somehow it is deleting the primary folder.. –  Webby Mar 21 at 3:56
    
@Webby crap, sorry, I fixed it, it will only delete the contents now. In any case, I just used that cause the * screwed with the code highlighting on the site, feel free to replace it with the rm -rf you had before. –  terdon Mar 21 at 3:59
    
Thanks that issue got fixed.. Now I am seeing very strange issue. I started running your script on our production machine. But somehow it's not even running, meaning its not copying any files and no error is getting thrown as well? So to debug this out I added echo statements echo $dir1 echo $dir2 and this echo statement got printed out fine with the right file path.. But I added this echo statement echo $primary_files echo $secondary_files and nothing got printed out. Do I need to do anything special if I need to use rsync? Because my earlier script was working fine. Any idea? –  Webby Mar 21 at 4:10
    
@Webby the rsync is printing nothing because of the 2>/dev/null, that will hide all error messages, you can remove that to see what it's printing. I tested this as best I could (I can't recreate your setup since it's quite complex) but echo $primary_files prints the list here. You can also try adding echo in front of the rsync command, that way you'll see what it's going to run. Let me know if that gives more info. –  terdon Mar 21 at 4:35
    
Sure make sense.. I just removed this 2>/dev/null and then I got error message on the console.. I have updated the question with that.. I have no idea what does that mean? But If I run my earlier shell script, it works fine. :( And I am running my shell script on machineA. –  Webby Mar 21 at 4:42

rsync takes care of: copying only the files that changed, ignoring files that you don't want copied (switch -C for example, which will exclude the same files that CVS would exclude in their repositories, although you can specify anything), recursively copying the entire structure of directories (of course, only the needed changes, not everything). It can optionally compress the stream which speeds up the transfer. It's also faster because it does the entire copying in a single connection.

Because you are copying only a single file, most of these features are not used. You would use

rsync -avz "$firstfile" "$secondfile"

which is exactly the same as scp except for the flags (a - archive keeps permissions and timestamps, v for verbosity and z for compression).

However, you could just as well use compression with scp:

scp -p -C …

I think this is the easiest solution here. Just add a single flag and you are done.

share|improve this answer
    
Thanks for suggesting. For your scp -C what does -C flag do? If it is compression, then will it affect how it is getting stored in machineA after copying the files in machineA or it is just for transfer process? –  Webby Mar 20 at 7:54
    
It is compression of the stream over the internet. It just reduces the bandwidth (of course it depends on your file, if it is already compressed, it won't help much). The final file is the same. The speed of rsync is also mostly because of compression and skipping unmodified files. There isn't much more a program can do when it is limited by your internet connection. –  orion Mar 20 at 7:57
    
Sure, let me try adding that -C flag to see whether it is some faster or not. –  Webby Mar 20 at 7:58
1  
That script isn't copying a single file, it has loops that copy one file at a time. –  Gilles Mar 20 at 23:05
    
If the files are on different systems, there isn't much you can do but keep the loop. If they are in the same system, you can very efficiently use rsync -- just synchronize the entire folder recursively, or use --include and --exclude switches. –  orion Mar 20 at 23:08
readonly TGT=/export/home/david/dist
readonly TGT1=${TGT}/primary
readonly TGT2=${TGT}/secondary
readonly MMAP_LOC=/data/pe_t1_snapshot
readonly PART1='t1_weekly_1680_[03579]_200003_5.data' # shell globbing does 
readonly PART2='t1_weekly_1680_[12468]_200003_5.data' # the bulk of the work
readonly F_LOC=BC
readonly SSH="david@machine"
#hoping the = works - I don't know 
SSH1='ssh -o "StrictHostKeyChecking=no" '"${SSH}${F_LOC%?}"
SSH2="${SSH1%?}${F_LOC#?}"  
DIR="${MMAP_LOC}/"'[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
DIR1="$($SSH1 'cd ${d='"$DIR"'} && echo $d')" #shell glob
DIR2="$($SSH2 'cd ${d='"$DIR"'} && echo $d')" #shell glob
${DIR1:?FAIL} [ -n "${DIR1#"$DIR2"}" ] && exit 1 #tests if d1=d2 or dies
F1="$($SSH1 'printf "%s\n" '"${DIR1}/${PART1}")" #prefers primary
F1="${F1}$(echo ; $SSH1 'for f in '"${DIR2}/${PART1}"'\ #shell glob in
        do { case "'"$F1"'" in "${f#'"$DIR2"'}") continue ;;\ # favor
                *) printf "%s\n" "$f" ;;\ #of files found in primary
        esac ; } ; done')" #with secondary as backup
F2="$($SSH2 'printf "%s\n" '"${DIR2}/${PART2}")" #secondary
rsync -avzt -e "${SSH1}:/" "${TGT1}"/. \ #if it works, based on your
    --exclude=* $(printf --include=%s\\n $F1) #file sizes, should
rsync -avzt -e "${SSH2}:/" "${TGT2}"/. \  #dramatically decrease
    --exclude=* $(printf --include=%s\\n $F2) #transfer times

Does this work?

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.