0

I want to scrape a forum by using an API and curl.

API:

https://api.hackertarget.com/pagelinks/?q=websitetotest.com

Here is the initial link:

https://bitcointalk.org/index.php?topic=840124.0

Desired outcome:

https://bitcointalk.org/index.php?topic=840124.20

I need to be able to do this all the way up to 4,240

https://bitcointalk.org/index.php?topic=840124.4240

As you can see, the only integer that changes is after the dot.

3

Curl, if that's what you want to use to access the URLs, supports URLs with ranges:

curl ... 'https://bitcointalk.org/index.php?topic=840124.[0-4240]'

Where ... are other options that you may want to use.

The range syntax also allows for stepping with another increment than 1. For example, to get every 20th URL:

curl ... 'https://bitcointalk.org/index.php?topic=840124.[0-4240:20]'

For further information, please refer to the curl manual.

To generate a list of URL for separate processing with curl or some other program, consider using a brace expansion (if your shell supports it):

printf '%s\n' 'https://bitcointalk.org/index.php?topic=840124.'{0..4240}

With bash and some other shells, an increment may be used as in '...URL...'{0..4240..20}.

The list could then be piped to e.g. xargs that could invoke your curl or other program with the URLs as arguments.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.