0

I have written a console application in .Net Core 2.0. it’s a process that consumes AMPQ messages with the RabbitMQ .NET client and passes them along. The program works great when I run it directly (i.e. “$ dotnet my_app.dll”), but it does not seem to have network access when I run it as a systemd service. I can see console output as it initializes and sets up RabbitMQ but it never gets to the Received event. I believe that the network access block is manifested as a channel-level exception in the RabbitMQ .NET client. What things could keep the systemd instance of the application from getting external network access (and cause a channel-level exception) that would not affect a direct execution?

I am experiencing this issue when the server has already been online for hours and I manually start the service ("$ sudo systemctl start my_app").

This is my myapp.service file:

[Unit]
Description=my_app
After=network.target

[Service]
WorkingDirectory=/opt/rh/rh-dotnet20/root/usr/lib64
ExecStart=/opt/rh/rh-dotnet20/root/usr/lib64/dotnet/dotnet /path_to_my_app.dll
Restart=on-failure
Environment="LD_LIBRARY_PATH=/opt/rh/rh-dotnet20/root/usr/lib64"

[Install]
WantedBy=multi-user.target

The server is Linux version 3.10.0-693.21.1.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Wed Mar 7 19:03:37 UTC 2018

Edit #1 4/26/18

I disabled SELinux and there was no change after boot.

The behavior is the same at boot and manual "systemctl start" much later.

Exceptions from the application are captured extensively, none arise. This is in line with my expectation that the source is a RabbitMQ client channel-level exception. I do not have access to the RabbitMQ server or it's logs at this time.

I have reviewed \var\log\audit\audit.log and The only entries I see related to this are SERVICE_START and SERVICE_STOP entries. When started manually, the are sequential (nothing else happened in the 30 seconds between systemd's start and stop).

I installed the application on a different CentOS 7 instance and experienced the same behavior: can run directly, can't run via systemd.

Nothing meaningful in /var/log/messages

In /var/log/secure I see the following two entries every time I start the service:

Apr 27 08:59:51 CHN-EventAuto polkitd[629]: Registered Authentication Agent for unix-process:1875:5129964 (system bus name :1.68 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) Apr 27 08:59:51 CHN-EventAuto polkitd[629]: Unregistered Authentication Agent for unix-process:1875:5129964 (system bus name :1.68, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

in netstat I see:

Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 chn-eventauto.lan:53430 ccssqlw01.lan.cle:mysql ESTABLISHED tcp 0 36 chn-eventauto.lan.c:ssh 10.10.200.51:62025 ESTABLISHED tcp 0 1 chn-eventauto.lan:52220 192.41.78.209:amqps SYN_SENT tcp 0 368 chn-eventauto.lan.c:ssh 10.10.200.51:61836 ESTABLISHED

192.41.78.209 is the IP of the server I am trying to connect to.

Edit #1 5/02/18 I was able to create the following tcpdump for port 5671: $ sudo tcpdump -i any port 5671 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 09:14:17.609095 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [S], seq 2553878816, win 29200, options [mss 1460,sackOK,TS val 414582912 ecr 0,nop,wscale 6], length 0 09:14:25.625102 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [S], seq 2553878816, win 29200, options [mss 1460,sackOK,TS val 414590928 ecr 0,nop,wscale 6], length 0 09:14:36.589776 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [S.], seq 2115926527, ack 2553878817, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1076644011 ecr 414575903], length 0 09:14:36.589901 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 1, win 457, options [nop,nop,TS val 414601892 ecr 1076644011], length 0 09:14:36.589917 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [S.], seq 2115926527, ack 2553878817, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1076644011 ecr 414576904], length 0 09:14:36.589959 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 1, win 457, options [nop,nop,TS val 414601892 ecr 1076644011], length 0 09:14:36.590097 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [S.], seq 2115926527, ack 2553878817, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1076644011 ecr 414590928], length 0 09:14:36.590276 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [S.], seq 2115926527, ack 2553878817, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1076644011 ecr 414578908], length 0 09:14:36.590394 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [S.], seq 2115926527, ack 2553878817, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1076644011 ecr 414582912], length 0 09:14:36.650356 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 1:320, ack 1, win 457, options [nop,nop,TS val 414601953 ecr 1076644011], length 319 09:14:36.692415 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 320, win 510, options [nop,nop,TS val 1076644114 ecr 414601953], length 0 09:14:36.695749 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 1:1449, ack 320, win 513, options [nop,nop,TS val 1076644117 ecr 414601953], length 1448 09:14:36.695775 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 1449, win 502, options [nop,nop,TS val 414601998 ecr 1076644117], length 0 09:14:36.696276 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 1449:2897, ack 320, win 513, options [nop,nop,TS val 1076644117 ecr 414601953], length 1448 09:14:36.696302 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 2897, win 547, options [nop,nop,TS val 414601999 ecr 1076644117], length 0 09:14:36.696490 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 2897:3258, ack 320, win 513, options [nop,nop,TS val 1076644117 ecr 414601953], length 361 09:14:36.696514 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 3258, win 592, options [nop,nop,TS val 414601999 ecr 1076644117], length 0 09:14:36.699425 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 320:446, ack 3258, win 592, options [nop,nop,TS val 414602002 ecr 1076644117], length 126 09:14:36.740911 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 446, win 512, options [nop,nop,TS val 1076644163 ecr 414602002], length 0 09:14:36.741437 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 3258:3484, ack 446, win 513, options [nop,nop,TS val 1076644163 ecr 414602002], length 226 09:14:36.781085 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 3484, win 638, options [nop,nop,TS val 414602084 ecr 1076644163], length 0 09:14:37.018333 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 446:483, ack 3484, win 638, options [nop,nop,TS val 414602321 ecr 1076644163], length 37 09:14:37.061116 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 483, win 512, options [nop,nop,TS val 1076644483 ecr 414602321], length 0 09:14:37.074754 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 3484:3999, ack 483, win 513, options [nop,nop,TS val 1076644496 ecr 414602321], length 515 09:14:37.074790 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 3999, win 683, options [nop,nop,TS val 414602377 ecr 1076644496], length 0 09:14:37.142794 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 483:938, ack 3999, win 683, options [nop,nop,TS val 414602445 ecr 1076644496], length 455 09:14:37.185490 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 938, win 509, options [nop,nop,TS val 1076644607 ecr 414602445], length 0 09:14:37.185921 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 3999:4048, ack 938, win 513, options [nop,nop,TS val 1076644607 ecr 414602445], length 49 09:14:37.185945 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 4048, win 683, options [nop,nop,TS val 414602488 ecr 1076644607], length 0 09:14:37.190889 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 938:987, ack 4048, win 683, options [nop,nop,TS val 414602493 ecr 1076644607], length 49 09:14:37.194567 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 987:1048, ack 4048, win 683, options [nop,nop,TS val 414602497 ecr 1076644607], length 61 09:14:37.231659 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 987, win 512, options [nop,nop,TS val 1076644653 ecr 414602493], length 0 09:14:37.241393 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 1048, win 512, options [nop,nop,TS val 1076644663 ecr 414602497], length 0 09:14:37.241676 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 4048:4090, ack 1048, win 513, options [nop,nop,TS val 1076644664 ecr 414602497], length 42 09:14:37.251377 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 1048:1090, ack 4090, win 683, options [nop,nop,TS val 414602554 ecr 1076644664], length 42 09:14:37.293200 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 1090, win 512, options [nop,nop,TS val 1076644715 ecr 414602554], length 0 09:14:37.293844 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 4090:4135, ack 1090, win 513, options [nop,nop,TS val 1076644716 ecr 414602554], length 45 09:14:37.299787 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 1090:1161, ack 4135, win 683, options [nop,nop,TS val 414602602 ecr 1076644716], length 71 09:14:37.341981 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 1161, win 512, options [nop,nop,TS val 1076644764 ecr 414602602], length 0 09:14:37.342961 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 4135:4207, ack 1161, win 513, options [nop,nop,TS val 1076644765 ecr 414602602], length 72 09:14:37.350771 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 1161:1233, ack 4207, win 683, options [nop,nop,TS val 414602653 ecr 1076644765], length 72 09:14:37.395374 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 1233, win 512, options [nop,nop,TS val 1076644817 ecr 414602653], length 0 09:14:37.398421 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 1233:1270, ack 4207, win 683, options [nop,nop,TS val 414602701 ecr 1076644817], length 37 09:14:37.401899 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 4207:5655, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 1448 09:14:37.401989 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 5655:5684, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 29 09:14:37.402004 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 5684, win 728, options [nop,nop,TS val 414602704 ecr 1076644823], length 0 09:14:37.402201 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 5684:6016, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 332 09:14:37.402638 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 6016:7464, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 1448 09:14:37.402650 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 7464, win 819, options [nop,nop,TS val 414602705 ecr 1076644823], length 0 09:14:37.403040 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 7464:8912, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 1448 09:14:37.403123 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 8912:8941, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 29 09:14:37.403133 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 8941, win 864, options [nop,nop,TS val 414602706 ecr 1076644823], length 0 09:14:37.403496 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 8941:9974, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 1033 09:14:37.403928 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 9974:11422, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 1448 09:14:37.403942 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 11422, win 954, options [nop,nop,TS val 414602706 ecr 1076644823], length 0 09:14:37.404336 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 11422:12870, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 1448 09:14:37.404433 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 12870:12899, ack 1233, win 513, options [nop,nop,TS val 1076644823 ecr 414602653], length 29 09:14:37.404444 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 12899, win 1000, options [nop,nop,TS val 414602707 ecr 1076644823], length 0 09:14:37.435689 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 1270:1325, ack 17777, win 1181, options [nop,nop,TS val 414602738 ecr 1076644823], length 55 09:14:37.442205 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 17777:19225, ack 1233, win 513, options [nop,nop,TS val 1076644863 ecr 414602653], length 1448 09:14:37.442592 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 19225:20673, ack 1233, win 513, options [nop,nop,TS val 1076644863 ecr 414602653], length 1448 09:14:37.442607 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 20673, win 1271, options [nop,nop,TS val 414602745 ecr 1076644863], length 0 09:14:37.442988 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 20673:22121, ack 1233, win 513, options [nop,nop,TS val 1076644863 ecr 414602653], length 1448 09:14:37.443474 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 22121:23569, ack 1233, win 513, options [nop,nop,TS val 1076644863 ecr 414602653], length 1448 09:14:37.443487 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 23569, win 1362, options [nop,nop,TS val 414602746 ecr 1076644863], length 0 09:14:37.443951 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 23569:25017, ack 1233, win 513, options [nop,nop,TS val 1076644863 ecr 414602653], length 1448 09:14:37.444339 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 25017:26465, ack 1233, win 513, options [nop,nop,TS val 1076644863 ecr 414602653], length 1448 09:14:37.444352 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 26465, win 1452, options [nop,nop,TS val 414602747 ecr 1076644863], length 0 09:14:37.444723 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 26465:27913, ack 1233, win 513, options [nop,nop,TS val 1076644863 ecr 414602653], length 1448 09:14:37.445154 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 27913:29332, ack 1270, win 512, options [nop,nop,TS val 1076644863 ecr 414602701], length 1419 09:14:37.445179 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 29332, win 1543, options [nop,nop,TS val 414602748 ecr 1076644863], length 0 09:14:37.445564 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 29332:30780, ack 1270, win 513, options [nop,nop,TS val 1076644864 ecr 414602708], length 1448 09:14:37.479662 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 30780:32228, ack 1325, win 512, options [nop,nop,TS val 1076644901 ecr 414602738], length 1448 09:14:37.479698 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 32228, win 1633, options [nop,nop,TS val 414602782 ecr 1076644864], length 0 09:14:37.484306 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 32228:33676, ack 1325, win 513, options [nop,nop,TS val 1076644905 ecr 414602747], length 1448 09:14:37.484726 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 33676:35124, ack 1325, win 513, options [nop,nop,TS val 1076644905 ecr 414602747], length 1448 09:14:37.484740 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 35124, win 1724, options [nop,nop,TS val 414602787 ecr 1076644905], length 0 09:14:37.485115 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 35124:36572, ack 1325, win 513, options [nop,nop,TS val 1076644905 ecr 414602747], length 1448 09:14:37.485550 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 36572:38020, ack 1325, win 513, options [nop,nop,TS val 1076644905 ecr 414602747], length 1448 09:14:37.485565 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 38020, win 1814, options [nop,nop,TS val 414602788 ecr 1076644905], length 0 09:14:37.485953 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 38020:39468, ack 1325, win 513, options [nop,nop,TS val 1076644905 ecr 414602747], length 1448 09:14:37.486382 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 39468:40916, ack 1325, win 513, options [nop,nop,TS val 1076644905 ecr 414602747], length 1448 09:14:37.486394 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 40916, win 1905, options [nop,nop,TS val 414602789 ecr 1076644905], length 0 09:14:37.486796 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 40916:42364, ack 1325, win 513, options [nop,nop,TS val 1076644905 ecr 414602747], length 1448 09:14:37.487216 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 42364:43812, ack 1325, win 513, options [nop,nop,TS val 1076644905 ecr 414602748], length 1448 09:14:37.487229 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 43812, win 1995, options [nop,nop,TS val 414602790 ecr 1076644905], length 0 09:14:37.487645 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 43812:45260, ack 1325, win 513, options [nop,nop,TS val 1076644905 ecr 414602748], length 1448 09:14:37.516357 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 45260:46708, ack 1325, win 513, options [nop,nop,TS val 1076644937 ecr 414602782], length 1448 09:14:37.516381 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 46708, win 2086, options [nop,nop,TS val 414602819 ecr 1076644905], length 0 09:14:37.517027 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 46708:48156, ack 1325, win 513, options [nop,nop,TS val 1076644937 ecr 414602782], length 1448 09:14:37.524263 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 48156:49604, ack 1325, win 513, options [nop,nop,TS val 1076644945 ecr 414602790], length 1448 09:14:37.524276 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 49604, win 2176, options [nop,nop,TS val 414602827 ecr 1076644937], length 0 09:14:37.524676 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 49604:51052, ack 1325, win 513, options [nop,nop,TS val 1076644945 ecr 414602790], length 1448 09:14:37.525093 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 51052:52500, ack 1325, win 513, options [nop,nop,TS val 1076644945 ecr 414602790], length 1448 09:14:37.525109 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 52500, win 2267, options [nop,nop,TS val 414602828 ecr 1076644945], length 0 09:14:37.525200 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 52500:53948, ack 1325, win 513, options [nop,nop,TS val 1076644945 ecr 414602790], length 1448 09:14:37.525272 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 53948:55396, ack 1325, win 513, options [nop,nop,TS val 1076644945 ecr 414602790], length 1448 09:14:37.525282 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 55396, win 2357, options [nop,nop,TS val 414602828 ecr 1076644945], length 0 09:14:37.525344 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 55396:56844, ack 1325, win 513, options [nop,nop,TS val 1076644945 ecr 414602790], length 1448 09:14:37.525404 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 56844:58292, ack 1325, win 513, options [nop,nop,TS val 1076644945 ecr 414602790], length 1448 09:14:37.525415 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 58292, win 2448, options [nop,nop,TS val 414602828 ecr 1076644945], length 0 09:14:37.525475 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 58292:59740, ack 1325, win 513, options [nop,nop,TS val 1076644945 ecr 414602790], length 1448 09:14:37.556633 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 59740:61188, ack 1325, win 513, options [nop,nop,TS val 1076644978 ecr 414602819], length 1448 09:14:37.556656 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 61188, win 2538, options [nop,nop,TS val 414602859 ecr 1076644945], length 0 09:14:37.556884 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 61188:62636, ack 1325, win 513, options [nop,nop,TS val 1076644978 ecr 414602819], length 1448 09:14:37.562046 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 62636:64084, ack 1325, win 513, options [nop,nop,TS val 1076644983 ecr 414602828], length 1448 09:14:37.562077 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 64084, win 2629, options [nop,nop,TS val 414602865 ecr 1076644978], length 0 09:14:37.562213 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 64084:65532, ack 1325, win 513, options [nop,nop,TS val 1076644983 ecr 414602828], length 1448 09:14:37.562304 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 65532:66980, ack 1325, win 513, options [nop,nop,TS val 1076644983 ecr 414602828], length 1448 09:14:37.562315 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 66980, win 2719, options [nop,nop,TS val 414602865 ecr 1076644983], length 0 09:14:37.562370 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 66980:68428, ack 1325, win 513, options [nop,nop,TS val 1076644983 ecr 414602828], length 1448 09:14:37.562438 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 68428:69876, ack 1325, win 513, options [nop,nop,TS val 1076644983 ecr 414602828], length 1448 09:14:37.562447 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 69876, win 2763, options [nop,nop,TS val 414602865 ecr 1076644983], length 0 09:14:37.562519 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 69876:71324, ack 1325, win 513, options [nop,nop,TS val 1076644983 ecr 414602828], length 1448 09:14:37.562593 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], seq 71324:72772, ack 1325, win 513, options [nop,nop,TS val 1076644984 ecr 414602828], length 1448 09:14:37.562660 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 72772:73957, ack 1325, win 513, options [nop,nop,TS val 1076644984 ecr 414602828], length 1185 09:14:37.563289 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [.], ack 73957, win 2718, options [nop,nop,TS val 414602866 ecr 1076644983], length 0 09:14:37.584981 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [P.], seq 1325:1396, ack 73957, win 2764, options [nop,nop,TS val 414602887 ecr 1076644983], length 71 09:14:37.625378 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 1396, win 512, options [nop,nop,TS val 1076645047 ecr 414602887], length 0 09:14:37.626254 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 73957:73998, ack 1396, win 513, options [nop,nop,TS val 1076645048 ecr 414602887], length 41 09:14:37.630428 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [F.], seq 1396, ack 73998, win 2764, options [nop,nop,TS val 414602933 ecr 1076645048], length 0 09:14:37.667716 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [.], ack 1397, win 513, options [nop,nop,TS val 1076645089 ecr 414602933], length 0 09:14:37.667978 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [P.], seq 73998:74029, ack 1397, win 513, options [nop,nop,TS val 1076645090 ecr 414602933], length 31 09:14:37.668008 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [R], seq 2553880213, win 0, length 0 09:14:37.668266 IP 192.41.78.209.amqps > chn-eventauto.lan.clearcallsolutions.com.47405: Flags [F.], seq 74029, ack 1397, win 513, options [nop,nop,TS val 1076645090 ecr 414602933], length 0 09:14:37.668281 IP chn-eventauto.lan.clearcallsolutions.com.47405 > 192.41.78.209.amqps: Flags [R], seq 2553880213, win 0, length 0

Edit #3 5/9/18 I tried the entire process on an Ubuntu 16.04 machine and experienced the same: app runs when manually started, will not run with systemd. I am thinking this is a quirk in dotnet and just not possible.

  • Please provide complete stack traces and other exception information. When your app is run from systemd, do you see AMQP connection establishment logged in the RabbitMQ logs? Do you see the connection using the ss or netstat commands? – Luke Bakken Apr 26 '18 at 16:13
  • Is this a problem after a systemctl start of your service, or only when the service is started at boot? Consider using After=network-online.target instead of network.target. – Mark Stosberg Apr 26 '18 at 16:47
0

A few assumptions:

  • When running the application manually, you have already enabled the rh-dotnet20 SCL with something along the line of scl enable rh-dotnet20 bash
  • This error does not explicitly call out the exact connection issue, so it could be an issue with a library or path of some kind.

My first thought here, is that something is setup in the SCLs activation, that is required for this to work properly. Try changing ExecStart to the following:

/usr/bin/scl enable rh-dotnet20 -- dotnet /path_to_my_app.dll

Also, you probably want to remove that LD_LIBRARY_PATH, since scl will configure it correctly for dotnet already.

Note: It may be possible to inspect the scl's activation script to determine what paths are needed for the service to start properly.

  • I tried that and there was no change. The process runs under scl[xxx] – Justin May 2 '18 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.