I was trying to run multiple machine learning experiments simultaneously to save time. My OS is Fedora 36, and I use emacs. To automate the experiment I used runs.py
import subprocess
subprocess.run("python train.py python train.py python train.py python train.py", shell=True)
where train.py
trains the neural network and I am just using different seeds.
Then, I open multiple shells inside emacs where in each I run python runs.py
. I can do at most 5 shells, when I open a sixth one and run python runs.py
, then emacs and the terminal close with no error. I am sure there is a log somewhere, but I don't know where. Why could this happen? train.py
uses the GPU, but there is enough memory, there is also enough RAM to run more, so I have no idea what's causing this.
I'd appreciate any help. I know the information I provided might not be enough, but I am not sure what else to say that could be potentially relevant.
emacs
from the command line (your terminal) and run the programs. All logs (stdout
andstderr
) should be printed in the terminal. Thus whenemacs
crashes you may see what error is causing a crash.emacs -nw &>/home/user/somefile
orstderr
only:emacs -nw 2>/home/user/errfile
systemctl status systemd-oomd
Killed due to memory used
. Thanks for your help! Now I know what needs to be fixed. My program saves intermediate results as it runs, so at some point I run out of memory. It wasn't obvious to me because at first there is enough memory, but it gets occupied quickly.