Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I using an Amazon EC2 server small instance and by default it allows 382 threads per process. However, I wish to run 10,000 threads in a single process. Recently, I learned that this can be done by either reducing the stack limit or increasing virtual memory.

To the best of my knowledge, virtual memory can be increased by increasing RAM or swap space in Linux. Increasing RAM is not an option. I know how to increase swap space, but I am not sure what the optimum size I should go for is.

What I am not clear on is, how do swap space and RAM work when I run threads? Is there some specific limit after which swap memory makes the threads slow? Is there an optimal amount of swap space I should use?:

share|improve this question
1  
What are you running that needs 10k threads? Can you find a solution that won't need that much? –  Karlson Feb 10 '12 at 15:32
1  
I am scaling a crawler to crawl 2M urls in one day. I can try to find a different approach that won't need that much threads but first I wish to understand disadvantages of using 10,000 threads. My prime focus is to utilize the server resources fully. –  dragosrsupercool Feb 10 '12 at 15:34
3  
10,000 threads isn't a good way to make things scale, it is a good way to make a server come to a crawl, more than 1 thread per CPU or Core is just going to make the server context switch and run slower not faster. –  Jarrod Roberson Feb 10 '12 at 15:44
2  
@ZJR If your concurrency is greater then number of available cores this fact won't help you because swapping context in and out from Disk is quite likely to bleed into wake up time for waiting threads. –  Karlson Feb 10 '12 at 15:49
1  
@dragosrsupercool I wouldn't do it this way. The responses you get are for the most part if not completely are self contained so keeping data within a single thread that sends a request and then waits for response is to me useless. I would use a thread pool to process data so when response arrive I just assign data to a thread and move on. –  Karlson Feb 10 '12 at 16:43
show 7 more comments

migrated from programmers.stackexchange.com Feb 10 '12 at 15:46

This question came from our site for professional programmers interested in conceptual questions about software development.

1 Answer

Under Normal circumstances System Administrators make SWAP either 1x or 2x memory. In your case I don't know if this will apply or even suffice.

Without having details of what you are running it is very difficult to know what would be the appropriate site of SWAP but in general whatever you run if it is swapped out to disk you will get performance degradation that can increase the time to run by orders of magnitude. So before you go putting 10k threads that will be swapping like crazy may be you should look at the configuration or design of your application and do something else.

So analyze your application and determine the number of active concurrent threads and may be you don't need to run this many.

SOLUTION APPROACH AS REQUESTED

When you establish a connection to a website or webserver you maintain a list of requests sent to that particular connection, since responses will be received in order. After a thread sends a request down to a server you wait in select() loop for data to become available. When that happens you receive the data and send the request info and the received data down to another thread for processing.

share|improve this answer
    
I got it. 10x!! –  dragosrsupercool Feb 10 '12 at 16:01
2  
the use 1x or 2x ram for swap number is way old. This should not be used anymore with modern computers. In some cases yes, but most machines you dont want gigabytes upon gigabytes of swap unless you need to hibernate. –  Patrick Feb 10 '12 at 16:06
1  
@Patrick You're right but unless someone goes through the trouble of analyzing exactly how much will be used, and out of hundreds of systems I deployed I remember this done once, carving 64GB of swap out of 1+TB of diskspace is an easy and simple thing to do. –  Karlson Feb 10 '12 at 16:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.