Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I want to back up my SSD using the Linux dd command, but I'm not sure how reliable that method will be. I think I read somewhere that dd does not check for or report errors, so obviously if true then it will be a deal breaker.
This will be the command:
sudo dd status=progress bs=512K if=/dev/nvme0n1 of=/media/d/ssd.img
So please explain how reliable the dd command can be for said use case.
And, are there any more reliable and/or easier alternative?
TLDR: Use ddrescue
It supports resume/continue capabilities, has automatic logs, and tons of other options. More at the ddrescue home page.
Example syntax:
ddrescue /dev/sde yourimagename.image sde.log
IF you want to (given your comment mentioning restoring) restore the image from the command above onto another drive of the same exact size:
ddrescue -f yourimagehere.image /dev/sde restore.logfile
Furthermore, it is faster than dd is -- at least it does look like it is when comparing speed of ddrescue and dd + pv.
dd is fast if you use bs=128k or something, to make read/write system calls in 128kiB blocks instead of the default 512-byte sectors. (About half of your CPUs L2 cache size is a good tradeoff between per-system-call overhead vs. the kernel's copy_to_user / copy_from_user cache hits). Piping it through pv obviously costs more CPU time, although if you run pv to have it attack to an already running process (looking at file-positions in /proc/<PID>/fdinfo) then it avoids that memory bandwidth cost. I assume ddrescue chooses a reasonable block size by default.
May 17, 2021 at 18:05
-S option you can make ddrescue even faster than plain dd, as well as being able to do some other useful things like pre-allocating the output file or bypassing the OS’s page cache.
May 17, 2021 at 18:08
dd status=progress shows what it's doing. You don't need pv for that. (And even on any Unix, you could just ls -l the output file whenever you want.)
May 17, 2021 at 19:03
dd has conv=sparse. IDK when that feature was added. It doesn't have a preallocate option; you have to use fallocate(1) and dd bs=128k conv=notrunc,sparse. Although I think sparseness detection works in units of the bs, so you might want a smaller block size like 16k. For a device that might have read errors, certainly ddrescue is a very good choice, but other than that you don't need it. If you know its options then by all means use it. Good point about tricks like sparsifying the output, especially if you just used fstrim on your FS before backup.
May 17, 2021 at 19:33
The original comment about 'not check for or report errors' probably came because by default 'dd' does not pad out bad reads, so for block-orientated devices not only the bad block but all subsequent blocks will be incorrect (because they're no longer aligned). As others have said, this is fixable using the 'conv=noerror,sync' option, which tells dd to ensure blocks are remain on block boundaries. It should be joined by a block-size setting matching the filesystem block size, which is often 4096 bytes but can be lower.
I would agree with Johan Myréen's comment about using file backup, because the granularity of the backup is much smaller - an error backing up one file doesn't necessarily affect the others. You could also use a file system that uses error-correction on the file data (such as zfs, btrfs, and some configurations of others), so at the very least you know when errors happen and hopefully can fix them.
Another way to detect bad backups would be to use a message digest hash code, e.g. 'sha256' on the raw device (unmounted!!) and on the dd backup file... they should of course be the same.
Finally, best practice in backups is never to rely on only one backup... keep a minimum of 2!
A disk cloning utility - such as clonezilla will make a compressed copy of your disk, including partition tables etc but omitting unallocated space and also unused space in well-known file systems such as ext4 etc.
Obviously it is much faster to only copy data and not the unused space.
If you suspect that your NVME storage cannot be relied upon for read operations, then perhaps you might want to consider software such as rsync to extract most/all of your data (in addition to a cloning operation).
It is quite OK for whole of disk. You could pipe it through compression, and unused SSD space should compress well. Optimal block sizing helps.
However, compression does not work well with encryption.
Further, copying file system contents will provide a performance benefit and naturally create contiguous files, and while we may not think this matters for SSD, every little bit helps and contiguous free space helps.
.. and yes the performance benefit comes from not copying the space, or even the cluster tips.
ddrescue shines for magnetic media, if your SSD has errors, no amount of retrying is going to help, and there is no platter or moving head to settle from another direction. But ultimately you need to compare entire images (or their hash digest is often considered close enough) to verify.
dd, as the name suggests, makes a disk image backup. If you have a 1 TB drive with 100 GB used, the disk image will still be 1 TB. The conv=sparse option exists for dd if you know there are zero-blocks, and ddrescue has the --sparse command.
However, for backup purposes, you may not need a raw disk image. Copying files with rsync or putting everything into a (possibly compressed) tarball is sufficient for most backup use-cases, and working on a file level with a filesystem can be nicer than directly working with a mounted image. rsync also has a convenient delta-transfer algorithm which only sends the differences in files for updating backups (not the same as incremental backups, for which you'll need additional software).
d explained as "disk", "drive", "data", "dump", "destroyer". How to get from dd to " disk image backup" is a mystery to me. If I didn't know dd, its name would suggest me nothing.
May 18, 2021 at 8:15
dd famously has many nicknames but I guess the way I wrote it the joke doesn't come across at all
Have a look at partclone (apt-get install partclone), I think it is what you are looking for.
Yes, dd is fine but not really necessary.
I just use "cat" plus pbzip2 to create compressed images, e.g.:
# cat /dev/nvme0n1 | pv | pbzip2 > myimage.img.bz2
This hypocritic answer should have probably just been a comment.
Use rsync. It's the tool for the job.
I think I read somewhere that dd does not check for or report errorsReally!!?? Looks like fake news, about a coreutil piece of software.ddworks well. But unless you are trying to build an exact image, I wouldn't call it a backup tool.