RH 7.5 Linux using curl with sftp to pull multiple files ending in a particular file extension

Question

I am in a Red Hat Linux 7.5 environment. I am trying to write a curl script to pull/receive any file from a folder that ends with a particular file extension.

Previously, I have written a script to push/send using curl with the ftps (NOT SFTP) option, login & password, full trace logging and 2>&1 for any other errors. I was able to script that because the filenames were local and able to grep'ed by some script logic.

Basically my curl statement to send is this (No, those are not real references in the string, I changed those values for privacy.):

curl --ftp-ssl -k --trace /bedrock/fred/logs/sendpdf.log -T /bedrock/fred/brontosaurussteak.pdf -u fflintstn:H4s$anC#0P ftp://bedrockpoobalodge.com:27Memebers/fredflintstone/pdffiles/ >> /bedrock/fred/logs/transfer.log

Where all the reference above is in a script and the references to the paths and whatnot are put into variables in a loop processing hundreds of files every hour on the hour. It looks something like this in the working script:

curl $curlFTPC $curlTRCMD\_$trFlCntr\_$cntnt.txt $curlSF/$file $curlAHaD &>> $prcssLgFl

It is a different matter altogether when you do not know the filenames you will be pulling/receiving.

I've looked at a lot of examples that loop based on an 'curl eval' like this:

for i in $(curl https://someserver.com/alpha/beta/gamma/  | grep pdf | awk -F "=" '{print $2}' | jq '.[].full_path' | awk -F '"' '{printf("https://someserver.com/alpha/beta/gamma/%s\n",$2)}')

But I just can't seem to get something like this to work to pull/receive under the same protocol and authentication. I am sure I am missing something obvious.

I'm open to a solution or even alternatives so long as it is script'able in bash and is not using Expect, can use sftp AND I can have a full trace and logging of stdin, stdout & stderr.

I'm not open to using 'Expect'.

Answer 1

wget for the rescue. wget designed for file transfer and logging.