Parallel running with only limited CPU cores

Question

I have a 12 cores server and I want to run a script which take a file as input argument and does processing on it. I want to use 8 core and no further to do parallel processing on 8 files at a time on 8 cores and then jump to next once the processing for 1 is complete.

I have tried doing it with xargs like so:

ls /data/paths/ | grep new | xargs -i -P 8 -n 1 bash main.sh {}

But when I see the core the CPU utilization is attached in the snapshot.

I have also tried doing with parallel as follows:

ls /data/paths/ | grep new | parallel -j 8 --no-notice bash main.sh {}

My condition that I have to keep it limited to 8 CPUs.

EDIT:

When I run ps -ef | grep main.sh it shows 11-12 processes instead of 8.

Answer 1

You can use taskset to set the cpu affinity of a process. The following will make sure that your xargs command only uses the first 8 virtual CPU cores:

ls /data/paths/ | grep new | taskset -c 0-7 xargs -i -P 8 -n 1 bash main.sh {}

Note that from man taskset:

A user must possess CAP_SYS_NICE to change the CPU affinity of a process.