1

I have a ZFS on a HDD with ZIL SLOG on an SSD.

If that's relevant, I have also a LARC cache on an SSD.

How can I reconfigure it to be sure that a failure of the SSDs won't cause data inconsistency (non-conformance to POSIX filesystem calls result rules, like intermixing content of two write() operations that one comes after another in a single thread)?

I want to ensure my PosgreSQL DB on ZFS does not become inconsistent if I restore a backup snapshot of the HDD without restoring the SSDs. (I do take measure for syncing PostgreSQL in such a way that (provided Postgre has no bugs) POSIX-correct filesystem warrants the DB does not become inconsistent.)

2 Answers 2

1

The ZIL is only suppose to contain uncommited writes to stable disks for a short period. If you had a power failure and a SSD failure at the same time, this could be a problem. But if the ssd failed while things were otherwise normal, zfs should just transition from the equivalent of raid write back to raid write through mode. Performance might drop, but nothing should be immediately corrupted.

The point of ZIL is to quickly write changes to non-volatile storage so that the application can be told quickly to continue. If the power failed before those also got written to stable storage (disk), they would be copied from ZIL to stable storage when the zfs volume was next mounted after power up.

The whole point of a filesystem snapshot is that you get a stable version of the filesystem to copy that is not being actively written to. This has nothing to do with ZIL, as the snapshot shouldn't be writable, so ZIL won't have any pending writes for it.

Having said that, postgreSQL might not be happy having a filesystem snapshot restored. Unless postgreSQL is also told to snapshot or pause right before the ZFS snapshot, the zfs snapshot could contain some partial postgreSQL writes, which could be a problem. You might want to ask a separate question about how to properly back up a postgreSQL database. (...unless someone else wants to cover that here.)

0

The SLOG can be thought of as independent of the dataset. What that means is that once your pg data has been flushed to disk, the dataset can be snapshot, and backed-up, and the snapshot can be restored (to the same pool and/or to a different pool) whether it has a log device or not.

If you intend to physically remove a log (SLOG) or cache (L2ARC) device from your pool, you should, of course, remove it logically first:

zpool remove [poolname] [logdevice|cachedevice]

(See man zpool-remove)

If you don't remove a SLOG properly, the pool may fail to import on the next reboot. Recovering from this can be fairly easy (if there's no unflushed data still in the SLOG) or difficult/impossible to do without accepting some corruption of your data. There's a reason why it's often recommended to add two SLOG devices as a mirrored pair, and that's to avoid exactly this problem - i.e. avoid having a single point of failure capable of corrupting your pool.


I'd still be making regular pg_dump backups (to another dataset with its own snapshot and backup schedule) because I think text dumps are more reliable than binaries - especially if the binary snapshot was made while the postgresql server was still running (the server may not have written everything in-memory to disk when the snapshot was taken...but shutting down the server will make it write everything that it needs to restart in the same state). Also because when it comes to important data, more backups are more better.

BTW, I wrote a simple postgresql backup script years ago that dumps everything, then the pg globals (roles, etc), then the schema for each database and table, and then the data (as COPY ... FROM) and then the data again as column inserts. I've been using variants of it for around 20 years now. I posted a version of it on ServerFault at What's the best way to automate backing-up of PostgreSQL databases? back in 2009.

That version probably needs a few minor tweaks (esp. to the DBS=( $($PSQL --list --tuples-only ...) ) line that gets the list of databases. And if the backup dir is a zfs dataset with its own snapshot schedule, you won't need the YMD sub-directories or the find ... -mtime +30 ... to delete old backups. Also, you won't need to pipe pg_dump or pg_dumpall into gzip, just use compression on the backup dataset.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .