I have a shell script that downloads files from a list using wget, and resumes automatically if there is any non-critical error (because of unstable WIFI during storms).

The problem is, I want to write to a filename.part file and then remove the .part extension once completed (overwriting and such are handled at that point). This works for simple urls like http://myserver.org/myfile.doc, but completely fails to guess a filename like http://myserver.org/index.php?file_id=foo. (my method would try to write to index.php.part instead of the desired result)

I can have wget get the "final" filename automatically and write to it, but that won't let me use the -O option to save with a different extension, it leaves no control.

So my question is, is there any standard way (or tool) to get the "final" filename in a download URL so I can write to a file with the same name but an added extension? (using -O in wget or -o in curl, such as wget $URL -O "$URL_GUESSED_FILENAME.part")

My tools are either wget or curl, no preference even if I currently use wget. Alternatively, if there's a way to do it in Python, I can accept that too.

  • 6
    It sounds like you're trying to reinvent wget -c. Additionally, I think no matter what you do, you will run into cases where you cannot "guess" the file name. There is simply too much power on the server side for you to know what file name a given URL will yield. HTTP isn't a file sharing protocol. It's an API. – Warren Young Sep 17 '12 at 5:21
  • I know it's not a file sharing protocol. That's why I am asking. – magnamouse Sep 17 '12 at 5:26
  • 2
    Can you not just rename the file after the download is complete, as the next step of your shellscript? – ire_and_curses Sep 17 '12 at 6:15

Wget may store http://myserver.org/index.php?file_id=foo into a server provided filename, e.g. foo.ext. Usually this requires the wget option --content-disposition. If you want to download to -O foo.ext.part you need to find out about it first. This can be done by downloading the http header first, it will contain the filename.

With wget you can get the corresponding header line with a command like this one:

wget --spider --server-response URL 2>&1 | grep -i content-disposition

You need some trivial parsing on the header to extract the filename. It is optional of course. If not present you are out of luck and wget will make up a filename based on index.php. Can just as well use that for -O then.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.