0

so I want to backup my system drive (full drive not just partition) every month using dd, in an external hard drive. So I have something like this in my crontab

0 9 1 * *   dd if=/dev/sda | gzip -c > /mnt/5E13119070E2D202/Backups/system_drive.backup.img.gz

And that works fine. But I am trying to figure out how to replace /dev/sda (system drive) with something which is persistent between reboots.

Using blkid (trimmed):

/dev/sda5: UUID="58141b62-72af-463c-a3c3-57d0b739c632" TYPE="swap" PARTUUID="c1b89110-05"
/dev/sda1: UUID="a97d9b38-e8a6-4cc2-9684-b7e579c1a990" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="c1b89110-01"
/dev/sdg1: BLOCK_SIZE="512" UUID="5E13119070E2D202" TYPE="ntfs" PARTUUID="000b4ae7-01"

Any ideas?

7
  • yes, /dev/disk/by-uuid/… Commented Aug 27, 2023 at 12:09
  • also, gzip is a very unfortunate choice as compressor here: it's algorithm is old, slow and not very good at compressing. You could use pigz which at least uses multiple threads to speed things up. But realistically, you want to use something like zstd -7 -T0 to achieve the same speed as gzip achieves while getting typically around twice the compression. Commented Aug 27, 2023 at 12:12
  • 2
    also, absolutely no need for dd; you can just gzip < /dev/disk/by-uuid/58… > /mnt/backup/system_drive.img.gzip (or, better, zstd -7 -T0 < /dev/disk/by-uuid/58… > /mnt/backup/system_drive.img.zstd) Commented Aug 27, 2023 at 12:14
  • Why would the name of your system drive change between reboots?
    – aviro
    Commented Aug 27, 2023 at 12:17
  • @MarcusMüller, he said he doesn't want to backup a specific partition, but the full drive. /dev/disk/by-uuid only has links to the partitions.
    – aviro
    Commented Aug 27, 2023 at 12:17

1 Answer 1

0

But I am trying to figure out how to replace /dev/sda (system drive) with something which is persistent between reboots.

So, you need a "world wide unique name" for your whole disk. Luckily, Linux has you covered.

  1. Find out the unique name for your drive by finding the right wwn-* entry in /dev/disk/by-id. You could just ls -l /dev/disk/by-id/wwn-*and look for sda, or do something like find /dev/disk/by-id -name 'wwn-*' -lname '*/sda'. Either way, you get a symlink like /dev/disk/by-id/wwn-0x1234cafe.
  2. in your script,
devlink=/dev/disk/by-id/wwn-0x1234cafe

gzip < "${devlink}" > /mnt/5E13119070E2D202/Backups/system_drive.backup.img.gz

as there's literally no advantage to using dd here. Quite the contrary! Instead inserting dd with its own block sizes and potential copy overhead, just let the compressor do its work directly on input.

I do recommend you do not use gzip for that, for two reasons:

  1. it's algorithm is old, slow and not very good at compressing.
  2. it's single-threaded, putting another bottleneck atop of the already slow algorithm

You could, instead of gzip, at least use pigz, which is multithreaded and does the same. But, while I have much respect for Adler and his compressors, it's been a couple of decades, and modern compressors are both faster and better at compressing; so this would fare better on both speed and compression ratio fronts

#!/bin/sh
devlink=/dev/disk/by-id/wwn-0x1234cafe
zstd -5 -T0 --force -- "${devlink}" > /mnt/5E13119070E2D202/Backups/system_drive.backup.img.gz
#    ^  ^    ^       ^  ^
#    |  |    |       |  |
#    \-------------------- -5: use a medium-low compression ratio. 
#       |    |       |  |      still better than `gzip --best`, typically, but also faster.
#       |    |       |  |      Possible values are 1 to 18, where you typically see
#       |    |       |  |      diminishing returns for values > 12.
#       |    |       |  |      
#       \----------------- -T0: use as many threads as you have CPU cores
#            |       |  |      
#            \------------ --force: because the input is not a regular file, zstd want
#                    |  |           to be explicitly told that, yes, we want this.
#                    |  |      
#                    \---- --: afterwards there's file names. Just for good measure.
#                       |      
#                       \- "${devlink}": input filename

Couple other remarks:

  • Running a backup on an actively, read/write mounted root file system: you can do that, yes, but it'll be in need of repair the very moment you try to restore it. Kind of a bad idea, hence. Do this from a live image, preferably, or sync; mount -o remount,ro ${root partition} / before, and mount -o remount,rw … after, at least. Yes, that will mess with the operation of your system while the backup is running. But you're not doing a backup to write "some backup somewhere for uncertain purposes", but to have a restorable, reliable state of your machine.
  • If you need to do this on a live system, you'd typically have a different approach. You'd use LVM or a snapshotting file system (ZFS, btrfs) for your root and data volumes instead, would do a snapshot, and back up that snapshot. If I guess correctly from your partial partition list, your system isn't even set up to use LVM, that's really annoying (nobody wants to deal with raw partitions in 2023! this isn't the 90s.) for you. Consider reinstalling your system to use either LVM or btrfs or ZFS.
  • You're writing your backup to an NTFS volume. Linux NTFS drivers are probably less reliable than they should be for backups. Also, ntfs-3g is slow! So, you'd crank up the compression ratio (your gzip must use -9 / --best, and your zstd would tend towards -12 to -14) in expense of compression speed, because you'd mostly be limited by write speed, not compression speed, anyways, and writing less makes things
    • more reliable, because fewer bits written means fewer chances for mistakes, and
    • faster, since fewer bits to write, less to wait
8
  • Have you created the ASCII tree in the comments with the details of each argument manually from scratch, or is there a tool that helps with that?
    – aviro
    Commented Aug 27, 2023 at 12:51
  • @aviro manually, but I've very much considered making this a quick python script Commented Aug 27, 2023 at 12:51
  • hi I tried the script. Seems good but I think it writes the output file to /dev/disk/by_id folder instead of the desired folder. Maybe I copied something wrong. The script is: #!/bin/sh devlink=/dev/disk/by-id/wwn-0x50000f0000000000 zstd -12 -T2 --force -- "${devlink}" > /mnt/5E13119070E2D202/Backups/system_drive.backup.img.gz and after running it, it creates a 0 size file in /mnt/5E13119070E2D202/Backups/system_drive.backup.img.gz and a 1.7gig file in /dev/disk/by-id/wwn-0x50000f0000000000.zst
    – vzografos
    Commented Aug 27, 2023 at 15:26
  • 1
    @vzografos you are right, from the man pages of zstd: "Unless --stdout or -o is specified, files are written to a new file whose name is derived from the source file name: o When compressing, the suffix .zst is appended to the source filename to get the target filename". So just add --stdout to the command. And you can delete the .zst file that was created.
    – aviro
    Commented Aug 27, 2023 at 21:48
  • ah right, my mistake Commented Aug 27, 2023 at 22:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .