0

I was trying to run multiple machine learning experiments simultaneously to save time. My OS is Fedora 36, and I use emacs. To automate the experiment I used runs.py

import subprocess

subprocess.run("python train.py python train.py python train.py python train.py", shell=True)

where train.py trains the neural network and I am just using different seeds.

Then, I open multiple shells inside emacs where in each I run python runs.py. I can do at most 5 shells, when I open a sixth one and run python runs.py, then emacs and the terminal close with no error. I am sure there is a log somewhere, but I don't know where. Why could this happen? train.py uses the GPU, but there is enough memory, there is also enough RAM to run more, so I have no idea what's causing this.

I'd appreciate any help. I know the information I provided might not be enough, but I am not sure what else to say that could be potentially relevant.

6
  • Try opening emacs from the command line (your terminal) and run the programs. All logs (stdout and stderr) should be printed in the terminal. Thus when emacs crashes you may see what error is causing a crash. Feb 2, 2023 at 6:25
  • I open it from the command line: 'emacs -nw'. But I can't see any logs because when it crashes, it closes the terminal automatically.
    – Schach21
    Feb 2, 2023 at 6:38
  • Oh I see, it crashes all process. Maybe redirecting output to a file might work: emacs -nw &>/home/user/somefile or stderr only: emacs -nw 2>/home/user/errfile Feb 2, 2023 at 6:41
  • 1
    I'm thinking your process may be killed by another one. Do you have oom killer? Try running: systemctl status systemd-oomd Feb 2, 2023 at 17:02
  • 1
    So, after all, it seems it might be a memory problem. I get Killed due to memory used. Thanks for your help! Now I know what needs to be fixed. My program saves intermediate results as it runs, so at some point I run out of memory. It wasn't obvious to me because at first there is enough memory, but it gets occupied quickly.
    – Schach21
    Feb 2, 2023 at 17:32

0

You must log in to answer this question.

Browse other questions tagged .