I noticed that I was not getting new backups on our DR so I took a look and saw that lsyncd was stuck on the server that sends to the DR via lsyncd. Our backup logs showed endless errors of "Waiting for child process to terminate." I eventually killed the lsyncd tasks since they were stuck and have since been unable to restart lsyncd. The backup to DR is configured through /etc/lsyncd.conf.
When I run status on lsyncd after I run 'systemctl start lsyncd.serivce' I get the following error:
[root@backupsteve bin]# systemctl status lsyncd.service ● lsyncd.service - Live Syncing (Mirror) Daemon Loaded: loaded (/usr/lib/systemd/system/lsyncd.service; disabled; vendor preset: disabled) Active: inactive (dead) Apr 3 17:22:19 backupsteve systemd: Started Live Syncing (Mirror) Daemon. Apr 3 17:22:19 backupsteve systemd: Starting Live Syncing (Mirror) Daemon... Apr 3 17:22:19 backupsteve sh: Error: failure getting absolute path of [/usr/local/cherbles/archive/plause/backups/dr551-prod/sftpdrop2/latestbackup/] Apr 3 17:22:19 backupsteve sh: Error: Cannot access source directory: /usr/local/cherbles/archive/plause/backups/dr551-prod/sftpdrop2/latestbackup/
I am able to browse to the ../dr551-prod/sftpdrop2/latestbackup just fine. and see the backups. It is just a link to the latest backup folder. This particular backup (dr551-prod) entry in the .conf file is towards the middle, but it errors on this one immediately for some reason. There are probably 20 servers in front of this. I do not have any instances of lsyncd running and I have stop/started it a good bit of times.