1

Trying to get wget to handle a sort of complicated situation. Basically I want it to only allow files/folders that fit this system:

https://example.com/top/a-file
https://example.com/top/a-file.ext
https://example.com/top/nested/directory/one.ext
https://example.com/top/another/nested/dir/two?maybe=paramsToo

I don't want it to follow paths like these:

https://example.com/
https://example.com/a-file
https://example.com/a-file.ext
https://example.com/top/
https://example.com/top/directory/
https://example.com/top/directory/one.ext
https://example.com/top/another/deeply/nested/direct/two?maybe=paramsToo

Basically, I only want it to handle this:

/top/:file
/top/nested/directory/:file
/top/another/nested/dir/:file

Those specific paths are allowed.

Within those paths, I want to also only allow specific files (.html, .js, .css, or no extension).

What I tried is this:

wget --no-parent -r -R .zip,.tar,.gz,.jpg https://example.com/top

This partially works, except in this situation: when it is currently evaluating this page:

https://example.com/top/nested/directory/one.ext

...and that page links here:

https://example.com/top/a-file

...the --no-parent option prevents it from going up to the parent. What I wanted the --no-parent option to do was just not allow going above /top, but instead it seems to be relative to the currently evaluating page, which doesn't make sense.

I know about -X to exclude specific folders/paths, but I don't know all the possible paths it may encounter so I don't know what to exclude.

So wondering how to accomplish this.

0

Depending on how it's redirecting you to https://example.com/top/a-file, have you tried using

    --max-redirect 0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.