0

When I'm connected via ssh and are parallel processes running and it loses the internet connection all parallel processes. When I reconnect I find the following message in log file:

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 12 in communicator MPI COMMUNICATOR 4 DUP FROM 0 
with errorcode 15.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
0:Terminate signal was sent, status=: 15
(rank:0 hostname: pid:2953):ARMCI DASSERT fail. ../../ga-5-4/armci/src/common/signaltrap.c:SigTermHandler():477 cond:0

Distribution

Description: Ubuntu 16.04.6 LTS Release:
16.04 Codename: xenial

How can I prevent this crash?

1
  • Perhaps run the process on the remote system under a tmux or screen session?
    – thrig
    Commented Aug 1, 2022 at 21:17

1 Answer 1

0

tmux is the solution here. the problem is that when the ssh connection closes it ends whatever bash session you were running. to solve this just type tmux and then type run the start running your program. you can then hit ctrl + b and then d to exit. this will leave your code running in a separate bash session and when you close ssh it will continue to run. you can run ssh user@ip ps a with user and ip replaced with the real ones, to see if the process is still running after you close the connection

3
  • Sadly neither tmux nor screener are installed on the server.
    – gvd
    Commented Aug 1, 2022 at 22:43
  • is there a reason you cannot install them?
    – brekker
    Commented Aug 2, 2022 at 3:05
  • I'm not have root privilegies. IT support is not available for the next couple of weeks. :(
    – gvd
    Commented Aug 2, 2022 at 9:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .