I have a Python script parallelized through GNU parallel which finds a certain result which I would like to output to a file, which I currently do through standard Python file IO. The issue is that I open this file in each parallelized thread, and the threads are stepping on eachother's toes while writing. I want to implement a FIFO semaphore, which I presume would have to be in the GNU parallel script, however I am unsure how to do that if I am accessing the file from within Python. My current script is:
#!/bin/bash
time parallel -j$(nproc) -N0 python3 ./polynomial_generator.py ::: {1..10} --progress echo {} >/tmp/out
-j1to avoid running multiple processes. Your job presumably does benefit from running multiple workers in parallel, however, so you'll want to add locking to the code that handles I/O in the Python script. Any locking (e.g., withflock(1)) from the shell or parallel would just prevent multiple workers from running at the same time, which is the same as using-j1.