Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am running scrapy on ubuntu server 13. And while it is crawling the network becomes very slow and it becomes difficult to even ssh into the system.

I think I can solve the problem by having one more network interface card connected to a different router that scrapy doesn't use. Is it possible? How?

share|improve this question
add comment

1 Answer

Generally, yes. A few notes that might be useful:

  1. If you have more than one interface, you can set up static routes that specify which interface should be used to get to a specific destination. CentOS networking docs have a decent guide. That lets you specify what interface the system uses to get to a specific outbound address.

  2. It sounds more like you want to be able to get in on an alternate interface. Most daemons, like ssh, let you configure which interface they listen on. For ssh, see the sshd_config ListenAddress configuration.

  3. Neither of those were you actual question, of course, which was if it was possible to configure an application to use a specific interface. That's also possible, if you use the iptables --uid-owner option to specify a route that's only used by apps from that user. There's a useful discussion of this scenario on linuxquestions.org .

  4. Finally, are you sure you're network bottlenecked? If you can't even get a prompt, I'd check to make sure scrapy isn't consuming too much CPU, and consider nice-ing it etc. if so.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.