Questions tagged [rsync]
rsync is a tool to efficiently copy directory hierarchies, locally or remotely, with powerful filters to decide what gets copied.
2,076
questions
0
votes
2
answers
120
views
Are Rsnapshot backups self contained - how to save into zip?
I have some old backups done by rsnapshot. The directories are:
/backups/week.0
/backups/week.1
/backups/week.2
/backups/week.3
/backups/week.4
I'd like to keep only one copy and delete the others. ...
0
votes
2
answers
74
views
Copy gitignored files from host A to host B (rsync?)
I have a git repo where several files containing sensitive info are gitignored. These files are located in various subdirectories at various depths.
Host A has all these files in place. I've cloned ...
0
votes
1
answer
184
views
Similar program to rsync but to instead use a shared cache directory or source?
With rsync, the files or data delta (difference) is compared directly between the source and destination.
Is there an alternative program that can compare the delta between the source, destination and ...
0
votes
1
answer
152
views
Copy new files from NAS to External Hard Drive with rsync
I want to copy new files from my NAS to my external hard drive. I did copy the new files manually up to this point. Now I want to use a script to automate this for me. However everything I tried so ...
-1
votes
1
answer
401
views
Why is rsync/scp slow from one server but fast from another?
I have 300 GB of large files located in server A. I am at home trying to copy those files, and both scp and rsync are painfully slow (80 KB/s or less).
I also have access to another server B. I can ...
2
votes
0
answers
151
views
Download tool to keep multiple ISOs up to date
I currently use this script (https://gist.github.com/Phill93/3c2b3503810a71bb98d6edb950d7aab2) to keep my ISO collection up to date. I use it to download ISOs via Rsync or HTTP/S. Now I would like to ...
0
votes
0
answers
16
views
Make Rsync log its decision about hardlink creation
This is an X-Post also asked on rsync mailing list.
I know it is a often discussed topic how rsync decide about using
hardlinks or copy a file. Even if content is unchanged problems are
often file ...
1
vote
1
answer
118
views
rsync include exclude is excruciatingly slow
When I use rsync to copy files matching a pattern à la https://unix.stackexchange.com/a/2503/288916 this works but it is excruciatingly slow. find locates the matching files much faster. (Like more ...
0
votes
0
answers
74
views
rsync between two machines with ssh and MFA without password
I have two machines that are accessible with ssh + MFA. I need to rsync regularly between these two but would like to avoid doing MFA each time. Setting up a keypair is not an option since MFA will ...
0
votes
1
answer
248
views
Rsync with date filter over ssh
I use the following command successfully
rsync -e 'ssh' -avr [email protected]:/home/mikrotik /bck/mikrotik/
How can I add date filter to this command? I would like to sync only files that are newer ...
0
votes
2
answers
54
views
rsync and delete other dir
My script has --delete --delete-after for rsync. I have a directory with name releases in same dir of basedir of rsync.
I don't want to delete releases directory after each running my script. Which ...
0
votes
0
answers
17
views
can rsync recognise files have been moved? [duplicate]
Scenario: I back up to external (ext) drive 1 and sync it to ext drive 2 (to have 2 backups). I archived a large directory (dir) on ext drive 1 by moving it into a new dir named 'archive'. Then i ...
0
votes
0
answers
403
views
NFS4 performance tuning for use with rsync
To open, I know that this is a bad idea, and should be avoided. However, we currently have no choice as our NFS server is a Netapp SAN, which means we can't just run rsync to the file server.
What I ...
3
votes
0
answers
182
views
MacOS files to NAS with rsync --iconv repeats the sync over and over
According to this hint and similar advice I am using the --iconv option in rsync (version 3.2.7) to sync file with umlauts (ä ü ö ...) to my Synology NAS. However the --iconv option does not work as ...
0
votes
2
answers
102
views
How to rsync multiple enumerated files
Suppose I have files data_1.mat, data_2.mat, ..., data_n.mat. How to I rsync those files to another folder in one line of code?
The only way I know is do it in a for loop:
for i in 1..n
do
rsync '...
1
vote
1
answer
44
views
Why does rsync always marks all folders as changed?
I'm on macOS Ventura and use rsync 3.2.7 to synchronize files between a NAS and a external hard drive.
When I run
rsync -avP --delete /Volumes/NAS/photos /Volumes/EXTERNAL_DRIVE
rsync marks all ...
1
vote
1
answer
298
views
rsync: excluding a directory from the source
It appears in the code below, rsync ignores --exclude. How would I correct that, and extend it to multiple directories?
$ rsync --help | grep Usage
Usage: rsync [OPTION]... SRC [SRC]... DEST
$ man ...
0
votes
0
answers
558
views
rsync permission denied - but they have full 777?
It is getting late and I am loosing my mind ...
I have a data drive (NTFS) which I used to share in an Windows/Ubuntu dual boot system. After an upgrade it is currently Windows 10 and Ubuntu running ...
0
votes
1
answer
105
views
rsync: [sender] out of memory: my_alloc caller (file=lib/pool_alloc.c, line=143)
I am trying to rsync a dir that includes a subdir with a ridiculous amount of files. So it was not a complete surprise when I got:
[sender] out of memory: my_alloc caller (file=lib/pool_alloc.c, line=...
0
votes
1
answer
135
views
Rsync script running manually but not via systemd service/timer
I've got a handful of bash scripts (all essentially clones of each other) which each does an rsync to a different dnf/yum repo to pull that repo down to our local mirror. The scripts all run correctly ...
0
votes
0
answers
30
views
I need to rsync from Ora1 vpn (as user michal) and rsync files into Ora2 vpn (as user srv_data). User 'michal' is in wheel grp on both Ora1 and Ora2
Hello Unix enthusiasts.
I have two vpms called Ora1 and Ora2.
I need to synchronize the contents of /DATA/ directory from Ora1, into the same /DATA/ directory located on the Ora2 vpm.
Ora1 /DATA/ ...
0
votes
0
answers
16
views
How to enforce rsync client-side transfer options via rsyncd daemon (such as --max-delete)?
Client is backing up with a QNAP which supports rsync to an rsyncd target remote storage I run. The QNAP has limited ability (esp via the GUI) to choose options for the rsync xfer.
I would like to ...
2
votes
2
answers
801
views
rsync to exclude everything except specific files
How to exclude all files except specific files with rsync?
I'm trying to only backup specific files with rsync, to exclude all files , except some specific files, namely the .md files, and solutions ...
2
votes
1
answer
389
views
Restore Rsync Backup Of System On Any Linux Distribution
I used the following alias to back up my root directory on Ubuntu 22.04 LTS to an external flash drive
alias backup='sudo rsync -aAXHS --info=progress2 --delete --exclude={"/dev/*","/...
0
votes
3
answers
197
views
Compare directories ignoring time differences of 1h +- 2s
I often want to synchronize my Music folders between my Android mobile phone and my Linux PC. The files on my phone are stored on an SD card using FAT32 and can come from either my Windows laptop or ...
0
votes
0
answers
178
views
rsync implications of ACL backups between different source & dest file systems
I'm having an issue using rsyncd (the daemon-ized version of rsync) to back up files from my macOS to my Synology server. Here's the command that fails:
% rsync -rlcAXtgoDiv --fake-super ~/scripts ...
0
votes
0
answers
43
views
Random rsync running in the background on Debian
I have recently had some lag in my system, when saving a file in vim.
I noticed that an rsync process was running. It was copying the ~/.openshot_qt dir (openshot is not currently installed on my ...
0
votes
1
answer
47
views
SCP from a distant server with a cumulative size limit
I want to copy a number of files from a distant server to local computer, such that the total size of the files does not exceed certain limit, as I do not locally have enough disk space; and the files ...
0
votes
2
answers
74
views
Methodology to copy data from one machine to many others in a fast way
I found some bottleneck on my workflow which is as follows. I do have one master computer which needs to send data to other node machines. This is done in a for loop such as:
for all nodes: rsync <...
1
vote
1
answer
31
views
Directory usage from remote server that can be accessible only via rsync
Need use directory from remote server that can be accessible only via rsync.
Can it be done without sync remote dir localy? (like sshfs)
Now this required so many attention just for some changed files ...
22
votes
3
answers
2k
views
rsync the file `a`b
Yeah, I know what you are thinking: "Who on earth names their file `a`b?"
But let us assume you do have a file called `a`b (possibly made by a crazy Mac user - obviously not by you), and you ...
0
votes
1
answer
359
views
How to copy directory with rsync to remote USB pen drive correctly
How to copy dir-tree form Debian 11 to remote pen drive in a way, that I can restore this directory to the OS without any permission modifications?
A usb pen drive is mounted to /home/<user>/mnt ...
0
votes
0
answers
99
views
Good security practice for remote to local backup using rsync
I would like to perform a manual backup of a remote VPS (Debian 12) to my laptop using rsync. I want to backup the single files and directories to easily inspect them locally, and not e.g. a VM ...
0
votes
1
answer
203
views
How do I show progress on rsync without --progress similar to dd SIGUSR1
How do I show progress on rsync without --progress similar to sending SIGUSR1 to dd?
When I accidentally start rsync without parameters on a large file and I neither pass --progress or --partial. Is ...
1
vote
1
answer
121
views
Would rsync overwrite the current files if I loose access to the files?
Would rsync overwrite to the current files that I have if I loose the access to certain data directories?
rsync -amHAXxzvR --update --numeric-ids --progress --include='*/' --include=*.sas7bdat --...
1
vote
1
answer
292
views
Rsync ignore files and folders by wildcards
I would like to ignore any files or folders under a specific folder when I am downloading from remove server with a foldername and wildcard.
I want to exclude all files or folders under "/*.old/&...
1
vote
1
answer
976
views
Moving root partition to a new drive, how do I mount both drives?
I currently have my root partition on a 480GB Sata SSD. I recently bought a new NVME 2TB drive which I would like to boot off of.
Using Arch Linux btw. What I plan to do is the following:
Create a ...
2
votes
2
answers
329
views
How to make rsync dst timestamp reflect time at which content was last transferred from src?
I am rsyncing a file from src to dst across a network. I’m navigating the many rsync options on the Linux man page, and need some help please.
A process on the destination responds to "newly ...
1
vote
1
answer
236
views
Transferring very large dataset from cluster to a storage server
We have to move a set of very large data (in petabytes) from HPC cluster to a storage server. We have a high capacity communication link between the devices. However, the bottleneck seems to be a fast ...
1
vote
1
answer
1k
views
How to determine which checksum `rsync` uses by default?
In the rsync image of output below:
I am unable to find the default checksum. Is it the first one,
or the one in brackets?
What is the bit-depth of just xxhash?
Found this DOCUMENTATION states the ...
0
votes
1
answer
93
views
How to Tranfser huge amount of Files to slow USB Drive (dd, cp, rsync, 7z, tar)
I have like 5GB of data and a very slow USB-Drive.
Should I use dd, cp or rsync?
Should I compress them first into for example 7z / tar or not?
In short: What is the best way and the best practices ...
1
vote
1
answer
778
views
Using rsync to query the checksum of a remote file
Can we use rsync to simply get the checksum of a remote file, without transferring it?
This could be useful for verifying the integrity of remote files without using a lot of bandwidth, if you don't ...
0
votes
0
answers
121
views
SSH server allows login but refuses connection when trying to copy a file
I am able to log into my ssh server both using a password and also with key. But when I try to copy a local file to the ssh server (rsync or scp) it refuses the connection which is very strange.
So ...
0
votes
0
answers
106
views
Why does transfer of a TAR file slow down so significantly mid-transfer?
I'm trying to transfer a 30GB TAR file between two servers. My problem is transfer starts at a fast rate of roughly 10s MB/s, but after 11% of the file transfer the speed slows enormously to 100s KB/s....
0
votes
1
answer
139
views
What to back-up in home partition?
The hard drive in the notebook I own is quite old. So, I have a habit of doing regular back-ups to avoid losing everything should it fail.
Rsync does fine job of backing-up the partition (it is a ...
1
vote
2
answers
780
views
Using sshpass with rsync to accept a fingerprint
I'm using rsync to copy a file to a device:
sshpass -p <password> rsync -avzP <file> <user>@<IP address>:<folder>
This has worked fine in the past but I was trying to ...
1
vote
2
answers
74
views
How to add file markers to check if script is already running
I wonder if I can get some help with a project I'm working on. I have a Synology NAS. I found a Community Package that autoruns a script of my creation anytime a USB drive is plugged in to one of the ...
0
votes
0
answers
253
views
Copy folder from one Linux server to another - cloning files onto a backup server
I would like to copy selected data files/folders from our a production server to a backup server. I want to do this every night at midnight. The servers are basically side-by-side, on the same ...
0
votes
2
answers
105
views
Copy only the folders (and all their content) at the level of source path with rsync
I have the following tree, where f are files while src and d are directories (f could be hidden or not):
src/f1
src/f2
...
src/d1
src/d2
src/d3
...
I want to replicate the same structure in a dst ...
2
votes
0
answers
265
views
How to make rsync skip existing files on destination
I have one Raspberry NAS and one backup server (192.168.25.25).
I would like to regularly backup the Raspebery NAS into the backup server where until now I have copied the files manually.
I drafted ...