4

I am using rsync to generate a list of files that have changed and then using that list to upload the files to s3:

rsync -av somefiles/ someotherfiles/ > list.txt


while read F  ; do
        echo $F
        aws s3 cp $lcDir/$F s3://durktest/blender/$F
done <list.txt

Example of what's in the list

swresample-2.dll
swscale-4.dll
tahoe.log
ucrtbase.dll
vcomp140.dll
vcruntime140.dll
2.78/
2.78/config/
2.78/config/bookmarks.txt
2.78/config/recent-files.txt
2.78/config/userpref.blend
2.78/datafiles/
2.78/datafiles/colormanagement/
2.78/datafiles/colormanagement/config.ocio
2.78/datafiles/colormanagement/filmic/
2.78/datafiles/colormanagement/filmic/filmic_desat65cube.spi3d
2.78/datafiles/colormanagement/filmic/filmic_false_color.spi3d
2.78/datafiles/colormanagement/filmic/filmic_to_0-35_1-30.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0-48_1-09.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0-60_1-04.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0-70_1-03.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0-85_1-011.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0.99_1-0075.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_1.20_1-00.spi1d
2.78/datafiles/colormanagement/luts/
2.78/datafiles/colormanagement/luts/aces_to_xyz.spimtx
2.78/datafiles/colormanagement/luts/adx_adx10_to_cdd.spimtx

Is there any better way to do this besides just using a fuse driver and doing rsync directly to s3?

  • I suppose, your somefiles/ someotherfiles/ means directory names. Do you want to just ignore subdirectories from copying? – RomanPerekhrest Jul 22 '17 at 6:54
  • I'd guess that would break on directory and file names containing line breaks. – n0rd Jul 22 '17 at 17:01
12

There is a better way to do this : use aws s3 sync

Example

aws s3 sync somefiles/ s3://durktest/blender

read AWS CLI Command Reference for more

7

To remove lines ending in /, just use sed:

sed '/\/$/d'  

That copies from standard input to standard output, deleting lines that end in /. Or use grep:

grep -v '/$'  

Probably lots of other ways.

5

With GNU xargs:

grep -v '/$' list.txt |
  xargs -rd '\n' -I@FILE@ aws s3 cp "$lcDir/@FILE@" s3://durktest/blender/@FILE@

(assuming $lcDir doesn't contain @FILE@)

Note that you have a few issues in your code, in that:

  • You called read without -r and without emptying $IFS
  • you forgot to quote $lcDir and $F
  • you used echo
  • in any case, you'll have problems with filenames containing newline characters with that approach.

If you did want to use a loop, you could do:

(
  ret=0
  while IFS= read -r F <&3; do
    case $F in
      (*/) ;;
      (*)
        printf '%s\n' "$F"
        aws s3 cp "$lcDir/$F" "s3://durktest/blender/$F" || ret=$?;;
    esac
  done 3< list.txt
  exit "$ret"
)    

You may also want to consider duplicity which AFAIK supports Amazon s3 as a backend.

  • what is this @FILE@? – cat Jul 22 '17 at 23:01
  • @cat, It's a placeholder: xargs reads in a line (filename) at a time, replaces the placeholder in the command with the line, and executes it. – deltab Jul 23 '17 at 2:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.