Scenario: Ftp site has 100s of files. Once a day, all the .gz files are copied to an AWS site then renamed. Only the date is kept in the file name.

Some of the files corrupted during copy. A txt file was delivered with a snapshot of ALL the files and attributes from the FTP site.

I have to write a script that will

  1. read the text file only for .gz files
  2. convert the name of the file to the filename on the S3 site
  3. Compare the file size from the txt file to the file size on the S3 server
  4. If the file sizes doen't match, write the filename and percent diff out to a txt file.

This is what I have so far, which is not even close to working


# Create array from files in FTP site text file
    f=cat ftpfiles.txt | grep .gz | awk '{print $9,$5}' #this doesn't work

# Start ForEach loop for files in created array
    for f in *.gz
    # Create variable for file size of source file
        file1size=$( cat $f | wc -c )

    # Create variable for file size of destination file
        fiesize2=aws s3 ls s3://folder1/folder2/$f | awk '{print $3}'

    #Compare sizes and print result to .txt file by percent diff
        echo "$f"

        if [ $file1size -lt $file2size ]; then

        dc -e "
        $( cmp -n $size -l $file1 $file2 | wc -l )

#close loop 

closed as unclear what you're asking by polym, maxschlepzig, jordanm, Braiam, Michael Homer Aug 6 '14 at 10:05

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.



f=cat ftpfiles.txt | grep .gz | awk '{print $9,$5}'


f=`cat ftpfiles.txt | grep .gz | awk '{print $9,$5}'`

That will fix the "This doesn't work" part of your script.


fiesize2=aws s3 ls s3://folder1/folder2/$f | awk '{print $3}'


file2size=`aws s3 ls s3://folder1/folder2/$f | awk '{print $3}'`

(Note that in addition to the quote marks, your original variable name is wrong here - fiesize2 compare with file2size)

Not the answer you're looking for? Browse other questions tagged or ask your own question.