Get total size of a list (containing HTTP links)

Question

I have a file containing a list of files on the web, and I would like to know the total files' size (in human-readable format) and check their availability. Is there a command/solution to do so?

P.S. I know I should use wget --spider but I don't know how to use it with a list.

Answer 1

You can use curl to get the size of a file on the web without downloading it (provided the webserver supplies this information.) The technique is detailed here.

Now, iterate over the list to get the size of each file:

cat files.lst | xargs -n 1 -I {} curl -sI {} | grep Content-Length | awk '{print $2}'

to get the list of sizes.

EDIT: You can of course use the awk to sum the sizes:

awk '{ sum+=$2 } END { print sum}'