2

Using wget, is it possible to supply more than one user/password pair when mirroring a website that has multiple password-protected areas with different credentials?

I would like to mirror an entire website that has several protected areas. As a simplified example, consider the following structure:

example.com/protected_area_1/ (user=alice, password=cat)
example.com/protected_area_2/ (user=bob, password=dog)

Now when using wget --mirror example.com, you can additionally supply the parameters --user and --password, but by supplying only one user/password pair, one of the two protected areas could not be downloaded due to 401 Authorization Required.

Is there a way to do something like wget --user=alice --password=cat --user=bob --password=dog (...)? In natural language: When encountering a password-protected area, try to gain access by trying all of the supplied credentials one after the other.

1 Answer 1

1

If this needs to be done with wget:

  1. If there is a small number of username/password combinations, and the download is not excessive, you can run the whole tree repeatedly with each combo (using --user and --password) and combine the result.

  2. If the different user/passwd combinations depend on domains, you can try wget's option --ask-password and expect, and provide the right one at the right time.

For more complex situations of downloading I always used a dedicated selenium based program, that correctly deals with current practise of loading part of the pages with javascript as well as with other login schemes than the basic authentication that wget can handle.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .