I upgraded my servers. Then I started corosync service one by one on my servers. I started first on 3 server and I wait 5 min. Then I started next 4 corosync on other servers and 7 server crashed in same time. I'm using corosync since 5 years. I was using;

Kernel: 4.14.32-1-lts
Corosync 2.4.2-1 
Pacemaker 1.1.18-1

and I never saw this before. I guess something is broken in new corosync version really really bad!

Kernel: 4.14.70-1-lts
Corosync 2.4.4-3 
Pacemaker 2.0.0-1

-

This is my corosync.conf: https://paste.ubuntu.com/p/7KCq8pHKn3/ Can you tell me how can I find the reason of the problem?

Sep 25 08:56:03 SRV-2 corosync[29089]:   [TOTEM ] A new membership (10.10.112.10:56) was formed. Members joined: 7
Sep 25 08:56:03 SRV-2 corosync[29089]:   [VOTEQ ] Waiting for all cluster members. Current votes: 7 expected_votes: 28
Sep 25 08:56:03 SRV-2 corosync[29089]:   [VOTEQ ] Waiting for all cluster members. Current votes: 7 expected_votes: 28
Sep 25 08:56:03 SRV-2 corosync[29089]:   [VOTEQ ] Waiting for all cluster members. Current votes: 7 expected_votes: 28
Sep 25 08:56:03 SRV-2 corosync[29089]:   [VOTEQ ] Waiting for all cluster members. Current votes: 7 expected_votes: 28
Sep 25 08:56:03 SRV-2 corosync[29089]:   [QUORUM] Members[7]: 1 2 3 4 5 6 7
Sep 25 08:56:03 SRV-2 corosync[29089]:   [MAIN  ] Completed service synchronization, ready to provide service.
Sep 25 08:56:03 SRV-2 corosync[29089]:   [VOTEQ ] Waiting for all cluster members. Current votes: 7 expected_votes: 28
Sep 25 08:56:03 SRV-2 systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Sep 25 08:56:03 SRV-2 systemd[1]: Started Process Core Dump (PID 43798/UID 0).
Sep 25 08:56:03 SRV-2 systemd[1]: corosync.service: Main process exited, code=dumped, status=11/SEGV
Sep 25 08:56:03 SRV-2 systemd[1]: corosync.service: Failed with result 'core-dump'.
Sep 25 08:56:03 SRV-2 kernel: watchdog: watchdog0: watchdog did not stop!
Sep 25 08:56:03 SRV-2 systemd-coredump[43799]: Process 29089 (corosync) of user 0 dumped core.

                                                      Stack trace of thread 29089:
                                                      #0  0x0000000000000000 n/a (n/a)
Write failed: Broken pipe


coredumpctl info
           PID: 23658 (corosync)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Mon 2018-09-24 09:50:58 +03 (1 day 3h ago)
  Command Line: corosync
    Executable: /usr/bin/corosync
 Control Group: /system.slice/corosync.service
          Unit: corosync.service
         Slice: system.slice
       Boot ID: 79d67a83f83c4804be6ded8e6bd5f54d
    Machine ID: 9b1ca27d3f4746c6bcfcdb93b83f3d45
      Hostname: SRV-1
       Storage: /var/lib/systemd/coredump/core.corosync.0.79d67a83f83c4804be6ded8e6bd5f54d.23658.153777185>
       Message: Process 23658 (corosync) of user 0 dumped core.

                Stack trace of thread 23658:
                #0  0x0000000000000000 n/a (n/a)

           PID: 5164 (corosync)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Tue 2018-09-25 08:56:03 +03 (4h 9min ago)
  Command Line: corosync
    Executable: /usr/bin/corosync
 Control Group: /system.slice/corosync.service
          Unit: corosync.service
         Slice: system.slice
       Boot ID: 2f49ec6cdcc144f0a8eb712bbfbd7203
    Machine ID: 9b1ca27d3f4746c6bcfcdb93b83f3d45
      Hostname: SRV-1
       Storage: /var/lib/systemd/coredump/core.corosync.0.2f49ec6cdcc144f0a8eb712bbfbd7203.5164.1537854963>
       Message: Process 5164 (corosync) of user 0 dumped core.

                Stack trace of thread 5164:
                #0  0x0000000000000000 n/a (n/a)

I cant find more log so I can't dig the problem.

  • Are the kernel, corosync, and pacemaker upgrades the only changes? – Jeff Schaller Sep 25 at 10:57
  • Also I have more upgrade changes but yes. These are the only changes. – Morphinz Sep 25 at 12:01
  • You have more changes, but these are the only changes? That does not make sense... – Kusalananda Sep 25 at 12:05
  • Hello kusalananda. Like I said I have upgraded my kernel. On arch-linux when you upgrade something thats mean a lot of packages will be upgraded same time. So yes it should be too much changes but I didnt change anything on my conf or etc. – Morphinz Sep 25 at 12:10
up vote 0 down vote accepted

After downgrade to "corosync 2.4.2-1" problem solved. Why you guys vote "-" for the topic? It was so clear and like you see it was corosync's fault or the arch builder.

What ever if you living the problem just downgrade and save your time.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.