1

Similar to this thread, I have a remote machine with 8 cores that I want to use for running scripts in parallel (1 script per core at a time).

However, I don't have multiple bash scripts but a single Python3 script that I want to run with different inputs. I tried parallel python3 -c main.py input*, parallel -j 100% python3 -c main.py ::: input*, and parallel python3 main.py input* but nothing worked.

The exact error message is:

parallel: Error: -g has been retired. Use --group.
parallel: Error: -B has been retired. Use --bf.
parallel: Error: -T has been retired. Use --tty.
parallel: Error: -U has been retired. Use --er.
parallel: Error: -W has been retired. Use --wd.
parallel: Error: -Y has been retired. Use --shebang.
parallel: Error: -H has been retired. Use --halt.
parallel: Error: --tollef has been retired. Use -u -q --arg-sep -- and --load for -l.

I don't understand how this is related to my input. I didn't use any of these options.

I'm fairly new and inexperienced with Unix and couldn't get it to work myself or with googling. Any help is appreciated. Do I have to write a shell-script to help me with that?

  • There are a lot of different versions of parallel out there, do you know which one you have? (parallel --version or man parallel should list it somewhere) – Wieland Dec 1 '16 at 9:55
  • It's GNU parallel 20141022 – CGFoX Dec 1 '16 at 10:12
  • Make sure you do not have a variable PARALLEL set in your environment. Check with echo $PARALLEL. Try preceding your command with PARALLEL= , as in PARALLEL= parallel python3 ... all on one line. – meuh Dec 1 '16 at 18:05
  • Good idea but unfortunately it's not the problem. No environmental variable PARALLEL is set and preceding PARALLEL= leads to the same error message. Can I somehow reinstall parallel? Does it even make sense? – CGFoX Dec 1 '16 at 18:59
2

The problem was actually with how parallel was installed on the remote machine (running the newest Ubuntu). I came across a thread solving my problem: Run sudo rm /etc/parallel/config after installation on Ubuntu to get rid of the config which caused my error messages.

The command I use to run my python script with different inputs in parallel is: parallel -j 100% python3 main.py ::: inputs*

Nevertheless, thanks to everyone who was helping!

| improve this answer | |
0

You can do that by assigning every script to specific core using taskset first you need to get the process id for your script with ps aux | grep <scriptname, input> or using pgrep -f <scriptname, input> then pass it to taskset like this taskset -pc 0 $pid, this will assign the process to core 1. for more info about taskset http://linuxcommand.org/man_pages/taskset1.html

| improve this answer | |
  • So I can't just use parallel and let it handle the assignment to the different cores? – CGFoX Dec 1 '16 at 9:50
  • i am not aware with parallel tool but the way that i mentioned will do the job but we will need to write small script to make this process auto – Wissam Roujoulah Dec 1 '16 at 9:53
  • It sounds like I have to manually assign the different jobs to different cores. And what happens if I assign multiple jobs to the same core? Will it run them one after another or will run it concurrently by switching between the threads (I don't want that)? – CGFoX Dec 1 '16 at 10:14
  • don't run two scripts in one core at the same time wait until it finish then start the other one, take a look into this it may help you with ordering unix.stackexchange.com/questions/242087/… – Wissam Roujoulah Dec 1 '16 at 10:28
0

I'm not sure where you are getting your error message from. Normally the syntax would be

parallel -j 100% python3 -c main.py ::: input*

For example, if you just have 2 files inputa, inputb this runs in parallel

python3 -c main.py inputa
python3 -c main.py inputb

If you have 8 cpu cores, and, say, 10 input* files, the -j 100% argument makes parallel run only 8 commands at once at the start. When one finishes, the next command will be run until all 10 are done. We rely on normal Linux scheduling to assign each command to a separate cpu.

| improve this answer | |
  • It still gives me the same error. I updated the question to now contain the full error message(s). – CGFoX Dec 1 '16 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.