Tagged Questions
0
votes
0answers
38 views
How to tune Unix (Ubuntu) for web crawling?
What are some ways to tune some Unix settings in the kernel(or elsewhere) for web crawling purposes? I am using multi-curl, and am using a multi-threading approach. I've set a higher limit for maximum ...
4
votes
2answers
1k views
Throttle the download speed of wget or curl while downloading
Is it possible to throttle (limit) the download speed of wget or curl ?
Is it possible to change the throttle value while it is downloading ?
