1

I have a Python script parallelized through GNU parallel which finds a certain result which I would like to output to a file, which I currently do through standard Python file IO. The issue is that I open this file in each parallelized thread, and the threads are stepping on eachother's toes while writing. I want to implement a FIFO semaphore, which I presume would have to be in the GNU parallel script, however I am unsure how to do that if I am accessing the file from within Python. My current script is:

#!/bin/bash
time parallel -j$(nproc) -N0 python3 ./polynomial_generator.py ::: {1..10}  --progress echo {} >/tmp/out
2
  • The easiest fix is to use -j1 to avoid running multiple processes. Your job presumably does benefit from running multiple workers in parallel, however, so you'll want to add locking to the code that handles I/O in the Python script. Any locking (e.g., with flock(1)) from the shell or parallel would just prevent multiple workers from running at the same time, which is the same as using -j1.
    – zackse
    Jan 20, 2020 at 20:30
  • Python locking is something I hadn't thought of. I am more interested in implementing it through GNU parallel, however, as I want to generalize this solution to some SAGE math code I am also working on. Jan 21, 2020 at 15:28

1 Answer 1

1

Change the python script so that it sends it output to stdout and then do:

parallel -N0 --progress --results res/ python3 ./polynomial_generator.py ::: {1..10}

The output will now be in res/*/*.

4
  • Ole, great to hear from you! This solution is good, except for the fact that it doesn't seem to be in parallel, which is something I would like, as my code benefits significantly from a parallel implementation. Could this be edited to work in parallel? Jan 21, 2020 at 15:08
  • What makes you think it is not run in parallel?
    – Ole Tange
    Jan 22, 2020 at 3:49
  • I'm not getting full CPU usage during execution and I was under the impression that I had to use the -j argument to specify for multiple (hyperthreads in this case), but maybe I am just misunderstanding. Jan 22, 2020 at 14:48
  • Additionally, the progress indicator says I am using only one CPU core, whereas I would like to use all 8 available to me. Jan 22, 2020 at 14:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .