2

I have a set of scripts that run constantly while Postgresql is up. I modified the ExecStop commands in my .service file so that a special script to stop those scripts and wait for them to complete will run before the database is shut down. This all works.

However, yesterday IT shut down the server, and the ExecStop commands weren't run. I know this because of some chaotic results in the scripts, and because the special script logs its output to a file, and that file wasn't updated yesterday.

Note: I just looked at the history of the root user, and I think IT ran

shutdown -h now

Would that have bypassed the systemctl setup?

Can I put something in the .service file to make sure that ExecStop is run at system shutdown, or do I need to create a special service that's tied to shutdown?

I do not think that this is a duplicate of "How to run a script with systemd right before shutdown", because I do not want to create a special service that duplicates my existing Postgresql service.

I am very sure that my special script didn't run, because it appends to a log file when it runs, and that log file was last appended to about a week ago, when I last ran 'systemctl stop my-service-name.service' .

Here's the service file:

[Unit]
Description=PostgreSQL 9.3 main database server
After=syslog.target
After=network.target

[Service]
Type=forking

User=postgres
Group=postgres

# Location of database directory
Environment=PGDATA=/db_tier1/main/

ExecStartPre=/usr/pgsql-9.3/bin/postgresql93-check-db-dir ${PGDATA} ; /bin/rm -f /db_tier2/PGSERVICE-PROD-ABORT
ExecStart=/usr/pgsql-9.3/bin/pg_ctl start -D ${PGDATA} -s -w -t 300
ExecStop=/bin/sh /var/lib/pgsql/bin/PROD/run_db_pre_stop.sh ${PGDATA} ; /usr/pgsql-9.3/bin/pg_ctl stop -D ${PGDATA} -s -m fast
ExecReload=/usr/pgsql-9.3/bin/pg_ctl reload -D ${PGDATA} -s

# Give a reasonable amount of time for the server to start up/shut down
TimeoutSec=600

[Install]
WantedBy=multi-user.target
  • I would suggest doing this as its own service. A system update is likely to blow away changes to postgresql's own .system file. – Philip Couling Feb 6 at 16:37
  • @PhilipCouling No, it shouldn't be a separate service (at least there's no reason for that.) Regarding modifying a service unit shipped by the system, systemd supports override units just for this purpose (see systemctl edit for some more details on that.) Oh and service units use a .service extension, I don't think there's such a thing as a .system file in systemd. – filbranden Feb 6 at 17:14
2

systemd will execute the ExecStop= of all services it stops during shutdown, so your script probably is being executed.

One common issue is that your service unit (in this case postgresql.service) doesn't have all dependencies set correctly, in which case other shutdown procedure (such as shutting down networking or unmounting filesystems) might happen in parallel with your script.

In order to make sure these stay up long enough for the ExecStop= script to complete, you should list them in the After= dependencies of your service. (The order on shutdown is the reverse of that on startup, so your service's stop will happen before the stop of those dependencies.) For instance, you most likely want to have After=network.target if you need networking in your stop script.

If you're writing to filesystems other than the root, use RequiresMountsFor= to ensure they'll be available during the procedure.

If you have journald and have persistent storage for it (I'd say that's likely), you can take a look at what it says, to see if it mentioned your ExecStop= script or some errors that happened while executing it.

For example, this command:

$ journalctl -u postgresql -b -1

It will list all logs for postgresql.service (assuming that's the exact name of your service unit) on the last boot of the machine (-b -1 means the boot before the current one.) So it should show you startup at boot and stopping it during shutdown. Hopefully that helps you troubleshoot it.

  • @Adina Hopefully this will help... If you have further questions, please share more details from your system, such as the .service unit, perhaps parts of the ExecStop= script, journalctl output etc. It's hard to tell you specifically what's wrong in your case without further details... Thanks! – filbranden Feb 6 at 17:12
  • I don't have a persistent journal. I can change that, but obviously that won't help with this issue. – Adina Feb 6 at 20:27
  • @Adina Even if you can't get the logs of that particular execution anymore, you might need to debug that service unit (perhaps on a non-production machine), so journal logs might come handy then. – filbranden Feb 6 at 20:58
0

When systemd starts services, it will also create a dedicated control group (cgroup for short) for each service. You can run something like ps ax -O cgroup:150 | grep system.slice to see the cgroups assigned to each service, although the listing will be rather wide.

Note that if you stop and restart postgresql without using systemctl (say, with pg_ctl for example), the restarted postgresql process will no longer be a member of the control group systemd created for the service. As a result, systemd will think that the service is now down.

Running ExecStop= actions at shutdown for services that are known to be already down would be useless, so systemd will skip them.

If you have a Database Administrator who is unwilling to learn systemctl commands, a separate management tool that will stop and start postgresql in the traditional way as it deems necessary, or a well-learned Pavlovian reflex to do so yourself as part of some database operations, you might consider changing the postgresql.service unit to:

Type=oneshot
RemainAfterExit=true

This will stop systemd from monitoring the actual state of the PostgreSQL database engine, but it will allow stopping and restarting the database engine in the traditional way at will without jeopardizing the ExecStop= actions at shutdown.

  • I always use systemctl to stop and start postgresql (well, except for a server start, which will start my service automatically.) – Adina Feb 6 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.