3

I have a 12 cores server and I want to run a script which take a file as input argument and does processing on it. I want to use 8 core and no further to do parallel processing on 8 files at a time on 8 cores and then jump to next once the processing for 1 is complete.

I have tried doing it with xargs like so:

ls /data/paths/ | grep new | xargs -i -P 8 -n 1 bash main.sh {}

But when I see the core the CPU utilization is attached in the snapshot.enter image description here

I have also tried doing with parallel as follows:

ls /data/paths/ | grep new | parallel -j 8 --no-notice bash main.sh {}

My condition that I have to keep it limited to 8 CPUs.

EDIT:

When I run ps -ef | grep main.sh it shows 11-12 processes instead of 8.

  • 1
    Your CPU utilization doesn't tell us anything about how many processes are started by xargs or parallel. The scheduler may well assign a process to various cores in its lifetime. – muru Jun 4 at 6:48
  • @muru when I run ps -ef | grep main.sh it shows 11 processes instead of 8. – nainometer Jun 4 at 7:01
  • That way you're counting both grep, and xargs at the least in addition to your script's processes. Possibly something else as well. – muru Jun 4 at 7:03
  • 1
    Please show info in image also as text (it is currently inaccessible). – ctrl-alt-delor Jun 4 at 7:59
  • 3
    Your utilisation is approximately 66.6% = ⅔ = 8 / 12 Therefore jobs is doing its job. However the OS, is balancing it. – ctrl-alt-delor Jun 4 at 8:01
3

You can use taskset to set the cpu affinity of a process. The following will make sure that your xargs command only uses the first 8 virtual CPU cores:

ls /data/paths/ | grep new | taskset -c 0-7 xargs -i -P 8 -n 1 bash main.sh {}

Note that from man taskset:

A user must possess CAP_SYS_NICE to change the CPU affinity of a process.

  • 1
    This is also true for GNU Parallel (taskset will even make it only detect 8 cores, so -P is not needed): ls /data/paths/ | grep new | taskset -c 0-7 parallel bash main.sh – Ole Tange Jun 8 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.