I have a Python script parallelized through GNU parallel which finds a certain result which I would like to output to a file, which I currently do through standard Python file IO. The issue is that I open this file in each parallelized thread, and the threads are stepping on eachother's toes while writing. I want to implement a FIFO semaphore, which I presume would have to be in the GNU parallel script, however I am unsure how to do that if I am accessing the file from within Python. My current script is:

time parallel -j$(nproc) -N0 python3 ./polynomial_generator.py ::: {1..10}  --progress echo {} >/tmp/out
  • The easiest fix is to use -j1 to avoid running multiple processes. Your job presumably does benefit from running multiple workers in parallel, however, so you'll want to add locking to the code that handles I/O in the Python script. Any locking (e.g., with flock(1)) from the shell or parallel would just prevent multiple workers from running at the same time, which is the same as using -j1. – zackse Jan 20 at 20:30
  • Python locking is something I hadn't thought of. I am more interested in implementing it through GNU parallel, however, as I want to generalize this solution to some SAGE math code I am also working on. – Alex Williams Jan 21 at 15:28

Change the python script so that it sends it output to stdout and then do:

parallel -N0 --progress --results res/ python3 ./polynomial_generator.py ::: {1..10}

The output will now be in res/*/*.

| improve this answer | |
  • Ole, great to hear from you! This solution is good, except for the fact that it doesn't seem to be in parallel, which is something I would like, as my code benefits significantly from a parallel implementation. Could this be edited to work in parallel? – Alex Williams Jan 21 at 15:08
  • What makes you think it is not run in parallel? – Ole Tange Jan 22 at 3:49
  • I'm not getting full CPU usage during execution and I was under the impression that I had to use the -j argument to specify for multiple (hyperthreads in this case), but maybe I am just misunderstanding. – Alex Williams Jan 22 at 14:48
  • Additionally, the progress indicator says I am using only one CPU core, whereas I would like to use all 8 available to me. – Alex Williams Jan 22 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.