Predict filename before downloading from a URL, in shell script

Question

I have a shell script that downloads files from a list using wget, and resumes automatically if there is any non-critical error (because of unstable WIFI during storms).

The problem is, I want to write to a filename.part file and then remove the .part extension once completed (overwriting and such are handled at that point). This works for simple urls like http://myserver.org/myfile.doc, but completely fails to guess a filename like http://myserver.org/index.php?file_id=foo. (my method would try to write to index.php.part instead of the desired result)

I can have wget get the "final" filename automatically and write to it, but that won't let me use the -O option to save with a different extension, it leaves no control.

So my question is, is there any standard way (or tool) to get the "final" filename in a download URL so I can write to a file with the same name but an added extension? (using -O in wget or -o in curl, such as wget $URL -O "$URL_GUESSED_FILENAME.part")

My tools are either wget or curl, no preference even if I currently use wget. Alternatively, if there's a way to do it in Python, I can accept that too.

Answer 1

Wget may store http://myserver.org/index.php?file_id=foo into a server provided filename, e.g. foo.ext. Usually this requires the wget option --content-disposition. If you want to download to -O foo.ext.part you need to find out about it first. This can be done by downloading the http header first, it will contain the filename.

With wget you can get the corresponding header line with a command like this one:

wget --spider --server-response URL 2>&1 | grep -i content-disposition

You need some trivial parsing on the header to extract the filename. It is optional of course. If not present you are out of luck and wget will make up a filename based on index.php. Can just as well use that for -O then.