2

I am looking for some direction - cause it sort of works, but doesn't.

I have a local hard drive and would like to copy ALL .JPG and .jpg files to a single directory on another local hard drive.

using rsync - my thinking: (source hard drive is a windows machine - file names have spaces)
-- get a list of all directory's on the source hard drive and put into a text file
-- edit dir list text file to add quotes around each dir
-- run rsync on each line from the text file using shell script loop



it works if I use rysnc at the command line and individually copy the dirctory.

rsync -r --include '*.jpg' --include '*.JPG' --exclude '*' --prune-empty-dirs /"media"/"tfrd"/"Disk06_01_M"/"Disk02_01_X"/"x_images and camera"/"vernon pics_other"/"office_PBWT4_YBP6D-7wmff_bpwg4_2vgby"/"office pro disk 2"/"BCM"/"Program Files"/"Microsoft Small Business"/"Business Contact Manager"/"SDKComponents"/"PPCRL"/ /home/tfrd/Desktop/filesync/rsync/dir2/  

Using Shell Script -- it isn't working - issues are with the dir variable and forward slash or spaces - and most of all -- it seems to try and parse prior to the last directory

I tried variation with the $IFS - because I think this is where the parsing issues is.


my script:

#!/bin/bash
file="/home/tfrd/Desktop/filesync/rsync/test_data_01.txt"

while IFS=$'\n' read -r line
#while IFS= read -r line
#while read line
do

#try 01
#rsync  --include '*.jpg' --include '*.JPG' --exclude '*/*/.' $line /home/rsync/dir2

#try 02
#rsync  --include '*.jpg' --include '*.JPG' --exclude '/*/' $line /home/rsync/dir2

#try 03
#rsync  --include '*.jpg' --include '*.JPG' --exclude '*/*/' $line /home/rsync/dir2

#try 04
rsync --protect-args --include '*.jpg' --include '*.JPG' --exclude '*/*/.' --prune-empty-dirs $line /home/rsync/dir2

done <"$file"



My test data: (just 2 lines in a txt file)

/"media"/"tfrd"/"Disk06_01_M"/"Disk02_01_X"/"x_images and camera"/"vernon pics_other"/"office_PBWT4_YBP6D-7wmff_bpwg4_2vgby"/"office pro disk 2"/"BCM"/"Program Files"/"Microsoft Small Business"/"Business Contact Manager"/"SDKComponents"/"PPCRL"/
/"media"/"tfrd"/"Disk06_01_M"/"Disk02_01_X"/"x_images and camera"/"vernon pics_other"/"office_PBWT4_YBP6D-7wmff_bpwg4_2vgby"/"office pro disk 2"/"SBA"/"program files"/"Microsoft Small Business"/"Office Accounting 2008"/"SDKComponents"/"PPCRL"/



Result: (cleaned up to be readable)

tfrd@Beagle-Ubuntu:~/Desktop/filesync/rsync$ clear


tfrd@Beagle-Ubuntu:~/Desktop/filesync/rsync$ ./script.sh 
/media/tfrd/Disk06_01_M/Disk02_01_X/x_images and camera/vernon pics_other/office_PBWT4_YBP6D-7wmff_bpwg4_2vgby/office pro disk 2/BCM/Program Files/Microsoft Small Business/Business Contact Manager/SDKComponents/PPCRL

rsync: link_stat "/media/tfrd/Disk06_01_M/Disk02_01_X/x_images" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/and" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//camera" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//pics_other/office_PBWT4_YBP6D-7wmff_bpwg4_2vgby" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/pro" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/disk" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//2/BCM" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//Files" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/Small" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//Business" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/Contact" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//Manager/SDKComponents" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0]


/media/tfrd/Disk06_01_M/Disk02_01_X/x_images and camera/vernon pics_other/office_PBWT4_YBP6D-7wmff_bpwg4_2vgby/office pro disk 2/SBA/program files/Microsoft Small Business/Office Accounting 2008/SDKComponents/PPCRL

rsync: link_stat "/media/tfrd/Disk06_01_M/Disk02_01_X/x_images" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/and" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//camera" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//pics_other/office_PBWT4_YBP6D-7wmff_bpwg4_2vgby" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/pro" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/disk" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//2/SBA" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//files" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/Small" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//Business" failed: No such file or directory (2)
rsync: link_stat "/home/tfrd/Desktop/filesync/rsync/Accounting" failed: No such file or directory (2)
rsync: change_dir "/home/tfrd/Desktop/filesync/rsync//2008/SDKComponents" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.0]
  • Is there some reason to use rsync specifically? And not just cp? – ilkkachu Dec 16 '17 at 22:58
  • @ilkkachu -- I am using rsync because I have LOTS of duplicated, and I want to do a binary compare plus date compare to make sure I only have one copy and the most recent. – David Dec 17 '17 at 2:25
  • I have used both solutions - ilkkachu and janos, and both solved the script issues. janos did it first. I am on to the next problem "why recursive is still happening" -- thanks for the help – David Dec 17 '17 at 2:26
  • though note that rsync mostly saves you just from useless transfers, I don't think it can detect identical files with another name. And by default it only checks the modification time and size; you'll need to use the --checksum flag to have it actually check the data. (And within a single host, if you do that, you might as well copy all the data to begin with.) If the timestamp if enough to check, GNU cp has the --update flag too. – ilkkachu Dec 17 '17 at 9:41
1

You don't need to quote every single path element.

In the text file with the list of paths, it will be better to not have any quotes, just the filenames as they are:

/media/tfrd/Disk06_01_M/Disk02_01_X/x_images and camera/vernon pics_other/office_PBWT4_YBP6D-7wmff_bpwg4_2vgby/office pro disk 2/BCM/Program Files/Microsoft Small Business/Business Contact Manager/SDKComponents/PPCRL/
/media/tfrd/Disk06_01_M/Disk02_01_X/x_images and camera/vernon pics_other/office_PBWT4_YBP6D-7wmff_bpwg4_2vgby/office pro disk 2/SBA/program files/Microsoft Small Business/Office Accounting 2008/SDKComponents/PPCRL/

And then write the script like this:

#!/bin/bash
file="/home/tfrd/Desktop/filesync/rsync/test_data_01.txt"

while IFS=$'\n' read -r line
do
    rsync --protect-args --include '*.jpg' --include '*.JPG' --exclude '*/*/.' --prune-empty-dirs "$line" /home/rsync/dir2
done < "$file"

That's it. Notice that $line is enclosed in "..." when used as the path argument of rsync. This will ensure that any embedded spaces and other special characters are handled correctly. The entire content inside the "..." is treated as a single value, that's why it works.

0

You had

while IFS=$'\n' read -r line ; do
    rsync ... $line /some/target/dir
done < inputfile

With inputfile containing lines like this:

/"media"/"tfrd"/"blah blah"/"yet another dir"/"foo"

You're somewhat on the right track in that using quotes and/or setting IFS will help against word splitting. However, there are some issues.

First, quotes inside a variable don't matter, at all, if the variable is split after expansion, it's split. E.g. var='"foo bar"'; printf "%s\n" $var will print "foo and bar" on two separate lines (they went to printf as two arguments). Instead, you need to quote the variable when expanding it, e.g. var='foo bar'; printf "%s\n" "$var".

Second, while the contents of IFS control word splitting, you're only setting it to the newline for the duration of the the read, not for the rest of the script. That saves you from read doing any splitting, but not from the splitting after the unquoted expansion. Setting IFS=$'\n'; at the start of the script would have made it effective for the expansion too, but it's still better to quote (because it avoids globbing too).

Since read takes newlines as delimiters to stop reading, we might as well give it an empty IFS, instead of one with newline.

So, you probably want to have

while IFS= read -r line ; do
    rsync ... "$line" /some/target/dir
done < inputfile

with the input file containing no quotes

/media/tfrd/blah blah/yet another dir/foo

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.