0

I have been using MPI in THCP diskless sever. However, when I run jobs with MPI, sometimes the process die because of SSH disconnected.

client_loop: send disconnect: Broken pipe

There is no error or no error in a single job.
Also if I tried to connect with SSH for each CPU after the job die, it connected normally

My source code use INTEL MKL library and server is composed with 16 cpus of AMD Ryzen 9 5900x and 48GB RAM memory, also I use MPICH3.

How can I solve this problem?
Is there a way to find the error log related to sshd disconnection uncertaintly?

Thank you.

Improve this question
3
  • A/ are you running systemd ? (is the journalctl utility available) ? B/ Is your diskless server /var/log directory mounted on tmpfs ? C/ (not immediately related) do all of your CPUs belong to the same NUMA node ?
    – MC68020
    Jun 24 at 5:44
  • Sorry for late response. 1. Yes, I can use journalctl 2. my diskless server mounted on /tftpboot/ (I created tftpboot directory in root) 3. I don't know NUMA, I just can access with SSH. Also, I use fedora35
    – JeongYeong Kim
    Jul 2 at 8:05
  • I suspect this to be caused by SSH Connection Timeout on server side. Check and if possible increase it : linuxhint.com/increase-ssh-connection-timeout On a side note, if all your cpus belong to the same numa node (let"s quickly say same hardware system) it's highly probable that MPI just don't bring any performance advantage.
    – MC68020
    Jul 2 at 15:47

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.