0

I have the following folder structure:

example.com
|
|_ folder1
| |_ file1.mp4
|
|_ folder2
| |_ file2.mp4
|
|_ folder3
| |_ file3.mp4
|
|_ folder4
  |_ file4.mp4

I would like to download all the mp4 files, which I can do with:

wget --no-parent -A mp4 example.com/

However, the files are large in size and it will not all fit on my machine. So instead, I just want to get the path of each mp4 and I can download when I have more space.

How can I get the following urls, and maybe output to a file (doesn't matter if url encoded or not):

example.com/folder1/file1.mp4
example.com/folder2/file2.mp4
...
1
  • I am not sure what you mean by "get". You could download the files to /dev/null, or you could just get the HTTP headers with curl --head -i /http://example.com.... May 23, 2021 at 1:44

2 Answers 2

0

You can make list of addresses utilizing wget:

# parse listing index.html 
$ wget -O- http://download.savannah.gnu.org/releases/lzip/ | grep -oE '["].*\.lz["]' | sed -e 's/^.*href=//' -e 's/\"//g' > urls.txt
# download files from list
$ wget -i urls.txt

If you have more of them you can put them in array and then

addresses=(\
"http://aaaaa" \
"http://bbbb" \
"http://ccc" \
)

for adr in ${addresses[@]}; do
    wget -O- ...
done

You can also combine all of it - make list of listings (index.html-s) in first step then list of .mp4-s in second and then download from that list with wget -i <urls>.

0

You can reject the mp4 files, while logging the rejections to a file that you can later use to download those mp4.

Based on your command above (you might want to add -r):

wget --no-parent --rejected-log=file.rej -R mp4 example.com/

The rejections will be in file.rej. It has many fields, the second one is the rejected URL.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .