1

So, I have 10 CPU core and 20 data to process. I want to process the data in parallel but I am afraid if I just process 20 at once it will make some problem. So, I want to process 10 data 2 times. Is there any command to do this?

Add info:

The data are in file format. It is quite huge, per file can reach 10GB. In my expreience if I launch more than 10 process, the PC will become really slow and even lag. So I am limiting the process to be only 10 which is equal to the number of cores. As for my RAM, I believe the software which process the file will not load everything at once so the RAM usage is quite low. That is why I just need to parallel the process for every 10 data. For now, I generate 10 shell script which execute parallel and each shell script contains sequential command.

Improve this question
4
  • Is the data in a file, in multiple files or is it already in RAM? If it's on disk, is the processing CPU bound or disk bound? If it's disk bound, would you be able to read the data efficiently 10 or 20 times in parallel? Could you tell us a bit more about what you have?
    – Kusalananda ♦
    Nov 7, 2018 at 6:36
  • I update the question
    – Bharata
    Nov 7, 2018 at 7:11
  • You have two (IMHO) perfectly good answers. How about choosing the one you like more, by selecting the green check mark just left of the text body. That way you will help others with the same interest as you go directly to that answer and you will be doing the community a service. Only you can choose the right answer to your question. Thank you.
    – Cbhihe
    Nov 9, 2018 at 12:02
  • well, to be honest they don't provide perfect answer. they just pointing out the softwre name and really basic usage. I found better explanation here opensource.com/article/18/5/gnu-parallel and techrepublic.com/article/…
    – Bharata
    Nov 12, 2018 at 5:15

2 Answers 2

Reset to default
3

Using GNU Parallel:

parallel my_process {} ::: files*

This will run one my_process file per CPU thread.

You can tell GNU Parallel to make sure there is 10G of RAM free before it starts the next job:

parallel --memfree 10G my_process {} ::: files*

If the free mem goes below 5G then GNU Parallel will kill the newest job and restart it when there is 10G free again.

Improve this answer
2
2

gnu parallel can do this. On Debian do apt-get install parallel. Then read the manual.

Gnu parallel works like xargs but does better with deciding whether to start more parallel processes.

(make can also do it.)

Improve this answer
1
  • can you elaborate more? I am unfamiliar with GNU parallel. So, let's say I want to process 20 file, file1.txt .... file20.txt. I will process this with a command line based software/program, dump-file. Usually, I just call: dump-file file1.txt. So, what should I do with GNU parallel?
    – Bharata
    Nov 8, 2018 at 3:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .