1

I am just trying to count how many URLs match with the pattern x--*.html. In this case below all files that match are being downloaded.

$ wget --random-wait --mirror --no-parent -A x--*.html myBaseURL &

Since I just want a count, is there some way to avoid transferring all these pages?

NOTE: I know there is a command --delete-after, but it still transfers.

Any ideas?

  • How is wget supposed to crawl a site without looking at the html pages to find all the links? – wurtel Sep 25 '14 at 14:00
  • Good question. What I mean is just do not download the url that matches. But you're right, it still will download, even in memory. --delete-after is the best option, do you agree? – coffee Sep 25 '14 at 14:12
  • 1
    You can tell wget to write the files to /dev/null. wget ... -O /dev/null. That way it won't even write them to disk, it'll just drop them from the get go. – slm Sep 25 '14 at 14:14
1

I do not believe there's an option to do this. In order to parse the contents of these files to see if there are additional links to follow, wget has to download them.

Your best option will likely be to do this:

$ wget --random-wait --mirror --no-parent -A x--*.html myBaseURL -O /dev/null &

To count the matches:

wget --random-wait --mirror --no-parent -A x--*.html myBaseURL -O | wc -l
  • How can I know how many files matched? – coffee Sep 25 '14 at 14:30
  • @coffee - see update. – slm Oct 17 '14 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.