2

I have 4 cores, and 4 python script files preprocess0.py, preprocess1.py, preprocess2.py, preprocess3.py. I would like to run these 4 processes in parallel using GNU parallel. I do not have input files. The input file is hardcoded inside each *.py file (it's read only, so it's ok). I would like to output the results to a file0.csv ... file3.csv files. This is as far as I've come:

parallel -j4 --progress python preprocess*.py ::: '>' ./file{}.csv

But it just stays there without writing anything, as if waiting for some input.

2

The syntax is:

parallel -j4 --progress 'python {} > ./file{}.csv' ::: preprocess*.py

That would create files called filepreprocess1.py.csv... You could use

parallel -j4 --progress 'python {} > ./file{#}.csv' ::: preprocess*.py

instead to use the job number instead and get some file1.csv... files. Or if you want to extract the number for the file names:

parallel -j4 --progress 'python {} > ./file{=s/[^\d]//g=}.csv' ::: preprocess*.py

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.