Hot answers tagged

390

To tar and gzip a folder, the syntax is: tar czf name_of_archive_file.tar.gz name_of_directory_to_tar The - is optional. If you want to tar the current directory, use . to designate that. To construct your filename, use the date utility (look at its man page for the available format options). For example: cd /var/www && sudo tar czf ~/www_backups/...


139

Since the servers are physically next to each other, and you mentioned in the comments you have physical access to them, the fastest way would be to take the hard-drive out of the first computer, place it into the second, and transfer the files over the SATA connection.


114

You can pipe tar to the split command: tar cvzf - dir/ | split --bytes=200MB - sda1.backup.tar.gz. On some *nix systems (like OS X) you may get the following error: split: illegal option -- - In that case try this (note the -b 200m): tar cvzf - dir/ | split -b 200m - sda1.backup.tar.gz. If you happen to be trying to split file to fit on a FAT32 ...


92

I would recommend that you just use the root account in the first place. If you set it up like this: Configure your sshd_config on the target machine to PermitRootLogin without-password. Use ssh-keygen on the machine that pulls the backup to create an SSH private key (only if you don't already have an SSH key). Do not set a passphrase. Google a tutorial if ...


79

Such an utility is zerofree. From its description: Zerofree finds the unallocated, non-zeroed blocks in an ext2 or ext3 file-system and fills them with zeroes. This is useful if the device on which this file-system resides is a disk image. In this case, depending on the type of disk image, a secondary utility may be able to reduce the size of the disk ...


73

Make use of the --rsync-path option to make the remote rsync command run with sudo privileges. Your command should then be e.g.: rsync -chavzP --rsync-path="sudo rsync" --stats user@192.168.1.2:/ . If sudo prompts you for a password, you have to avoid that either by using a NOPASSWD privilege with the remote user account (pointless for root, may make sense ...


69

It depends on whether the disk image is a full disk image, or just a partition. Washing the partition(s) If the disk is in good working condition, you will get better compression if you wash the empty space on the disk with zeros. If the disk is failing, skip this step. If you're imaging an entire disk then you will want to wash each of the partitions on ...


69

netcat is great for situations like this where security is not an issue: # on destination machine, create listener on port 9999 nc -l 9999 > /path/to/outfile # on source machine, send to destination:9999 nc destination_host_or_ip 9999 < /dev/sda # or dd if=/dev/sda | nc destination_host_or_ip 9999 Note, if you are using dd from GNU coreutils, you ...


66

As this question has many different answers, the following list should combine the suggestions into one comprehensive list: Under most circumstances you want to backup these: /home/ for user data and configuration. /etc/ for system wide configuration files. /var/ contains a mix of directories you usually want to backup and those you don't want to backup. ...


47

from man rsync: -t, --times preserve modification times EDIT - to improve on this answer since it is not immediately obvious why this did not help OP: OP is copying files from one filesystem to another and wanting to preserve c-time. Most people understand c-time to mean "create time" which is incorrect on most UNIX/Linux systems (...


47

The manual says: -I, --ignore-times don't skip files that match size and time With more details: -I, --ignore-times Normally rsync will skip any files that are already the same size and have the same modification timestamp. This option turns off this "quick check" behavior, causing all files to be updated.


33

It sounds like you want etckeeper from Joey Hess of Debian, which manages files under /etc using version control. It supports git, mercurial, darcs and bazaar. git is the VCS best supported by etckeeper and the VCS users are most likely to know. It's possible that your distribution has chosen to modify etckeeper so its default VCS is not git. You should ...


33

Summary of the methods (as mentioned in this question and elsewhere) to clear unused space on ext2/ext3/ext4: Zeroing unused space File system is not mounted If the "disk" your filesystem is on is thin provisioned (e.g. a modern SSD supporting TRIM, a VM file whose format supports sparseness etc.) and your kernel says the block device understands it, you ...


33

Do use fast compression. Whatever your transfer medium - especially for network or usb - you'll be working with data bursts for reads, caches, and writes, and these will not exactly be in sync. Besides the disk firmware, disk caches, and kernel/ram caches, if you can also employ the systems' CPUs in some way to concentrate the amount of data exchanged per ...


31

Package management is one of the main differentiators between distributions. Between unrelated distributions, you won't be able to do anything automatic. Different distributions break down software into different sets of packages and use different names. Between machines running the same version of the same distribution, you can achieve a similar ...


31

Only if you… select one of the delete options. See man rsync for more information, but here is an excerpt: --delete delete extraneous files from dest dirs --delete-before receiver deletes before xfer, not during --delete-during receiver deletes during the transfer --delete-delay find deletions during, delete after --...


30

I believe you can use rsync to do this. The key observation would be in needing to use the --existing and --update switches. --existing skip creating new files on receiver -u, --update skip files that are newer on the receiver A command like this would do it: $ rsync -avz --update --existing src/ dst Example Say ...


25

There are several limitations that could be limiting the transfer speed. There is inherent network overhead on a 1Gbps pipe. Usually, this reduces ACTUAL throughput to 900Mbps or less. Then you have to remember that this is bidirectional traffic and you should expect significantly less than 900Mbps down. Even though you're using a "new-ish router" are you ...


25

The rsync program does exactly that. From the man page: It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for ...


24

You can do this with the commands from the acl package (which should be available on all mainstream distributions, but might not be part of the base installation). They back up and restore ACL when ACL are present, but they also work for basic permissions even on systems that don't support ACL. To back up permissions in the current directory and its ...


22

Cron will attempt to send an email with any output that may have occurred when the command was run. From cron's man page: When executing commands, any output is mailed to the owner of the crontab (or to the user specified in the MAILTO environment variable in the crontab, if such exists). Any job output can also be sent to syslog by using the -s ...


21

Using rsync is fairly safe on read-write mounted file systems. When rsync is started it builds up a file list and then starts to copy those files. This file list is not being updated during the run. The actual data is then copied. This means when a file changes after rsync has built the file list, it will copy the new content. However, when a new file is ...


21

Why not use rsync instead? It's made for the job! rsync -uan --progress --exclude=".*" <source> <destination> The above will list all the files to be archived without actually copying anything. Check that the list is correct, then run it again with the n option removed in order to copy the files (you could also remove the --progress for a ...


19

If you have a multi-core machine using pigz is much faster than traditional gzip. pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data. pigz was written by Mark Adler, and uses the zlib and pthread libraries. Pigz ca be ...


18

Here is your solution: rsync -nrv --delete dirA/ dirB/ Instead of making the two folders identical, we use rsync to only show what it would do. That is the effect of -n. Careful, do not forget to add this option! The -r means a recursive scan, the -v gives the wanted verbose listing. You can add another -v to get all equals listed, too. The --delete ...


18

The add-apt-repository command is part of the software-properties-common package in Debian 8.x (jessie). Once you've installed software-properties-common, you'll have add-apt-repository available and you can use it to add PPA repositories. Note: Be sure that the packages you're installing are compatible with Debian. If the packages are only available for ...


17

One simple way to do this is to use the graphical ssh-askpass program with sudo, this gets around the fact that sudo is not connected to the terminal and allows you to enter the password safely: rsync -chavzPe 'ssh -X' --stats \ --rsync-path='SUDO_ASKPASS=/usr/lib/ssh/x11-ssh-askpass sudo -A rsync' \ user@192.168.1.2:/ . Of course the ssh-askpass ...


17

We deal with this regularly. The two main methods we tend to use are: SATA/eSATA/sneakernet Direct NFS mount, then local cp or rsync The first is dependent on whether the drive can be physically relocated. This is not always the case. The second works surprisingly well. Generally we max out a 1gbps connection rather easily with direct NFS mounts. You won'...


16

Try archivemount root@srv1:/backup# archivemount windows-2003-S.gz /target/ Unrecognized archive format root@srv1:/backup# archivemount -o formatraw windows-2003-S.gz /target/ Calculating uncompressed file size. Please wait. root@srv1:/backup# ls /target/ data root@srv1:/backup# file /target/data /target/data: DOS/MBR boot sector; partition 1 : ID=0x7, ...


16

If the image is read-only you can also use nbdkit (man page) and its xz plugin (xz should provide better compression and random access times than gzip). Create the compressed partition image dd if=/dev/sda1 bs=16M | xz -9 --block-size=16MiB - > sda1.img.xz A --block-size option of 16 MiB should provide good random access performance. Note: you may use ...


Only top voted, non community-wiki answers of a minimum length are eligible