Remove lines of text that end in “/”

Question

I am using rsync to generate a list of files that have changed and then using that list to upload the files to s3:

rsync -av somefiles/ someotherfiles/ > list.txt


while read F  ; do
        echo $F
        aws s3 cp $lcDir/$F s3://durktest/blender/$F
done <list.txt

Example of what's in the list

swresample-2.dll
swscale-4.dll
tahoe.log
ucrtbase.dll
vcomp140.dll
vcruntime140.dll
2.78/
2.78/config/
2.78/config/bookmarks.txt
2.78/config/recent-files.txt
2.78/config/userpref.blend
2.78/datafiles/
2.78/datafiles/colormanagement/
2.78/datafiles/colormanagement/config.ocio
2.78/datafiles/colormanagement/filmic/
2.78/datafiles/colormanagement/filmic/filmic_desat65cube.spi3d
2.78/datafiles/colormanagement/filmic/filmic_false_color.spi3d
2.78/datafiles/colormanagement/filmic/filmic_to_0-35_1-30.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0-48_1-09.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0-60_1-04.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0-70_1-03.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0-85_1-011.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_0.99_1-0075.spi1d
2.78/datafiles/colormanagement/filmic/filmic_to_1.20_1-00.spi1d
2.78/datafiles/colormanagement/luts/
2.78/datafiles/colormanagement/luts/aces_to_xyz.spimtx
2.78/datafiles/colormanagement/luts/adx_adx10_to_cdd.spimtx

Is there any better way to do this besides just using a fuse driver and doing rsync directly to s3?

Answer 1

There is a better way to do this : use aws s3 sync

Example

aws s3 sync somefiles/ s3://durktest/blender

read AWS CLI Command Reference for more

Answer 2

To remove lines ending in /, just use sed:

sed '/\/$/d'  

That copies from standard input to standard output, deleting lines that end in /. Or use grep:

grep -v '/$'  

Probably lots of other ways.

Answer 3

With GNU xargs:

grep -v '/$' list.txt |
  xargs -rd '\n' -I@FILE@ aws s3 cp "$lcDir/@FILE@" s3://durktest/blender/@FILE@

(assuming $lcDir doesn't contain @FILE@)

Note that you have a few issues in your code, in that:

• You called read without -r and without emptying $IFS
• you forgot to quote $lcDir and $F
• you used echo
• in any case, you'll have problems with filenames containing newline characters with that approach.

If you did want to use a loop, you could do:

(
  ret=0
  while IFS= read -r F <&3; do
    case $F in
      (*/) ;;
      (*)
        printf '%s\n' "$F"
        aws s3 cp "$lcDir/$F" "s3://durktest/blender/$F" || ret=$?;;
    esac
  done 3< list.txt
  exit "$ret"
)    

You may also want to consider duplicity which AFAIK supports Amazon s3 as a backend.