I am in a Red Hat Linux 7.5 environment. I am trying to write a curl script to pull/receive any file from a folder that ends with a particular file extension.

Previously, I have written a script to push/send using curl with the ftps (NOT SFTP) option, login & password, full trace logging and 2>&1 for any other errors. I was able to script that because the filenames were local and able to grep'ed by some script logic.

Basically my curl statement to send is this (No, those are not real references in the string, I changed those values for privacy.):

curl --ftp-ssl -k --trace /bedrock/fred/logs/sendpdf.log -T /bedrock/fred/brontosaurussteak.pdf -u fflintstn:H4s$anC#0P ftp://bedrockpoobalodge.com:27Memebers/fredflintstone/pdffiles/ >> /bedrock/fred/logs/transfer.log

Where all the reference above is in a script and the references to the paths and whatnot are put into variables in a loop processing hundreds of files every hour on the hour. It looks something like this in the working script:

curl $curlFTPC $curlTRCMD\_$trFlCntr\_$cntnt.txt $curlSF/$file $curlAHaD &>> $prcssLgFl

It is a different matter altogether when you do not know the filenames you will be pulling/receiving.

I've looked at a lot of examples that loop based on an 'curl eval' like this:

for i in $(curl https://someserver.com/alpha/beta/gamma/  | grep pdf | awk -F "=" '{print $2}' | jq '.[].full_path' | awk -F '"' '{printf("https://someserver.com/alpha/beta/gamma/%s\n",$2)}')

But I just can't seem to get something like this to work to pull/receive under the same protocol and authentication. I am sure I am missing something obvious.

I'm open to a solution or even alternatives so long as it is script'able in bash and is not using Expect, can use sftp AND I can have a full trace and logging of stdin, stdout & stderr.

I'm not open to using 'Expect'.

  • IMHO this: ftp://bedrockpoobalodge.com:27Memebers/fredflintstone/pdffiles/ is not valid URL – Romeo Ninov May 30 '19 at 18:52
  • No, those are not real references in the string, I changed those values for privacy. – ajtorre27 May 30 '19 at 18:54
  • 1
    try using wget instead of curl -- it seems to support mget / globbing (look at this page -- it could be as simple as wget --user=user --password=password ftps://host/path/*.pdf). I cannot give you an answer because I'm not able to test anything right now. – mosvy May 30 '19 at 19:03
  • 1
    @mosvy : Can wget be used in a script and can allow logging? – ajtorre27 May 30 '19 at 19:03
  • 1
    you seem to confuse sftp (ftp-imitating file transfer over ssh) with ftps (ftp + ssl, just like https is http + ssl). Your examples are clearly using the latter. If that's really the case, yes, wget supports logging (with the -o option) and can be used from a script ;-) – mosvy May 30 '19 at 19:07

wget for the rescue. wget designed for file transfer and logging.

  • 1
    Can you give an example usage showing how this could help the OP? – roaima May 30 '19 at 22:16
  • @dudi-boy Using sftp? Can you provide a usage example? – ajtorre27 May 31 '19 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.