2

I am just trying to count how many URLs match with the pattern x--*.html. In this case below all files that match are being downloaded.

$ wget --random-wait --mirror --no-parent -A x--*.html myBaseURL &

Since I just want a count, is there some way to avoid transferring all these pages?

NOTE: I know there is a command --delete-after, but it still transfers.

Any ideas?

Improve this question
3
  • How is wget supposed to crawl a site without looking at the html pages to find all the links?
    – wurtel
    Commented Sep 25, 2014 at 14:00
  • Good question. What I mean is just do not download the url that matches. But you're right, it still will download, even in memory. --delete-after is the best option, do you agree?
    – coffee
    Commented Sep 25, 2014 at 14:12
  • 1
    You can tell wget to write the files to /dev/null. wget ... -O /dev/null. That way it won't even write them to disk, it'll just drop them from the get go.
    – slm ♦
    Commented Sep 25, 2014 at 14:14

1 Answer 1

Reset to default
2

I do not believe there's an option to do this. In order to parse the contents of these files to see if there are additional links to follow, wget has to download them.

Your best option will likely be to do this:

$ wget --random-wait --mirror --no-parent -A x--*.html myBaseURL -O /dev/null &

To count the matches:

wget --random-wait --mirror --no-parent -A x--*.html myBaseURL -O | wc -l
Improve this answer
2
  • How can I know how many files matched?
    – coffee
    Commented Sep 25, 2014 at 14:30
  • @coffee - see update.
    – slm ♦
    Commented Oct 17, 2014 at 18:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .