0

I've had a look at this (and the forum thread here) and this.

I've tried running in Python and also at the command line. I've double-checked: some files have definitely been deleted from the source, but are present in the link-dest destination. I've tried messing around with numerous options. I've tried adding forward slash to the end of the paths to see if that might make a difference. The paths in all cases are simple directories, never ending in glob patterns. I've also looked at the man pages.

Incidentally, this shouldn't matter, but you never know: I'm running this under WSL (W10 OS).

Nothing seems to work.

By the way, the files deleted in source do get deleted (or rather not copied) in the target location (if not a dry run).

What I'm trying to do is to find out what changes have occurred between the link-dest location and the source, with a view to cancelling the operation if nothing has changed. But to do that I have to be able to get a list of new or modified files and also files which have been deleted.

This is the Python code I've been trying:

link_dest_setting = '' if most_recent_snapshot_of_any_type == None \
    else f'--link-dest={most_recent_snapshot_of_any_type[0]}'
rsync_command_args = [ 'rsync', 
                       '-v', 
                       # '--progress',
                       # '--update', 
                       '--recursive', 
                       '--times', 
                       '--delete', 
                       # '--info=DEL', 
                       '-n', 
    link_dest_setting, source_dir, new_snapshot_path, ]
print( f'running this: {rsync_command_args}')    
result = subprocess.run( rsync_command_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
rsync_result_stdout = result.stdout.decode( 'utf-8' )
print( f'rsync_result stdout |{rsync_result_stdout}|')
rsync_result_stderr = result.stderr.decode( 'utf-8' )
print( f'rsync_result stderr |{rsync_result_stderr}|')

Typical stdout (with dry run):

rsync_result stdout |sending incremental file list
./
MyModifiedFile.odt

sent 1,872 bytes  received 25 bytes  3,794.00 bytes/sec
total size is 6,311,822  speedup is 3,327.27 (DRY RUN)

|

(no errors are reported in stderr)

Just found another option, -i. Using this things get quite mysterious:

rsync_result stdout |sending incremental file list
.d..t...... ./
>f.st...... MyModifiedFile.odt

sent 53,311 bytes  received 133 bytes  35,629.33 bytes/sec
total size is 6,311,822  speedup is 118.10
|

Edit

Typical BASH command:

rsync -virtn --delete --link-dest=/mnt/f/link_dest_dir /mnt/d/source_dir /mnt/f/destination_dir

Dry run which, in principle, should show files/dirs present under link_dest_dir but NOT present (deleted) under source_dir. I can't get this to be shown. In any event I think the Python answer is likely to be a preferable solution, because the scanning STOPS at the first detection of a difference.

Edit 2

(in answer to roaima's question "what are you saving?")
My "My Documents" dir has about 6 GB, and thousands of files. It takes my Python script 15 s or so to scan it, if no differences are found (shorter if one is). rsync typically takes about 2 minutes to do a copy (using hard links for the vast majority of the files). If that were found to be unnecessary, because there had been no change between the source and the link-dest location, I would then have to delete all those files and hard links. The deletion operation on its own is very expensive in terms of time. Incidentally, this is an external HD, spinning plates type. Not the slowest storage location ever, but it has the limitations it has.

Just as importantly, because rsync does not appear to be capable, at least according to what I have found, of reporting on files which have been deleted in the source, how would I even know that this new snapshot was identical to the link-dest snapshot? In these snapshot locations I only want to keep a limited number (e.g. 5) snapshots, but I only want to add a new snapshot when it is different to its predecessor. So although the script may run every 10 minutes, the gap between adjacent snapshots may be 40 minutes, or much longer.

I see you (roaima) have a high rep, and seem to specialise quite a bit in rsync. The simple question I want answering is: is it possible for rsync, on a dry run or not, to report on files/dirs deleted in the source relative to the link-dest? If not, is this a bug/deficiency? Because the man pages certainly seem to claim (e.g. with --info=DEL) that this should happen.

7
  • I'm not great with python. What's the actual rsync command you're running? What do you want it to do, and what does it seem to be doing? Can you provide some example files (or directory structure) and explain what you want to happen with it? Commented Feb 25, 2021 at 13:38
  • What's mysterious about the -i output? It's, if anything, easier to parse in a script than the -v output.
    – Kusalananda
    Commented Feb 26, 2021 at 7:36
  • @Kusalananda Thanks, but I don't understand it. Can you explain what these dots and "f", "d", "t" and "s" mean? Preferably with a link to some documentation somewhere. Ideally I would like the names of the files absent in the link-dest location to be displayed. Commented Feb 26, 2021 at 8:03
  • The d and the t means ./ is a directory and that its timestamp is different. The f, s and t means the ODT file is a file, has a size difference and a timestamp difference. This is described in the rsync manual (see tho -i or --itemize-changes option in there).
    – Kusalananda
    Commented Feb 26, 2021 at 8:36
  • 1
    OK, the answer to that is long enough that I have to give it in an edit! Commented Feb 26, 2021 at 9:56

2 Answers 2

0

The key parts of your question seem to be these,

I only want to add a new snapshot when it is different to its predecessor. So although the script may run every 10 minutes, the gap between adjacent snapshots may be 40 minutes, or much longer

and

Is it possible for rsync, on a dry run or not, to report on files or directories deleted in the source relative to the link-dest

What I'm understanding here is that you have implemented a version of rsnapshot, where each time you consider taking a backup, the previous backup directory becomes your --link-dest directory. The big difference being that if there are no changes between the current source tree and the most recent backup, then in your case no backup is to be taken.

In my short experiment it seems to be possible simply to look at the output of rsync: if there is output there is work to be done, and if not there isn't. The key though is to look directly at the link directory

output=$(rsync -rti --delete --dry-run "$src/" "$lnk/" 2>&1 | grep -v '^[^*]d' | head -n1)
if [ -n "$output" ]
then
    # Work to be done
    rsync -rtiv --link-dest "$lnk/" "$src/" "$dst"
fi

At the moment I've omitted directories from the test, so a change to a directory will not trigger a backup. If you care about changes to directories too, remove the grep filter from the test

5
  • Thanks. In Linux I've used Timeshift, indeed. Actually it's not exactly like you describe, in the sense that although I compare with the previous snapshot of the particular time category (10-minutes, hour, day, week, month, etc.) for the purpose of identifying differences, I actually use the most recent snapshot in any time category for the non-dry rsync op ... since it seems to me that using the most recent snapshot will result in the most hard links and the fewest file copy operations. Will do some experiments with your solution later. Commented Feb 26, 2021 at 12:08
  • Hmmm... ! Sorry to say so, but when I modify or add a file in the src location (i.e. somewhere in the directory tree), this results in a non-dry run, work to be done. But if I delete a file in the src somewhere in the tree, output is indeed a blank string! Can you possibly see whether you have different results in your system? I'm really puzzled whether this is just my system or whether rsync seemingly fails to identify deletions as (it would appear from the docs) it is intended to. Commented Feb 26, 2021 at 19:50
  • I'm so sorry; I missed a final update to my grep when transferring from my test rig to here. Answer modified for you Commented Feb 26, 2021 at 20:58
  • Thanks, working now. So the conclusion seems to be that the reporting of deletions (and additions and modifications, presumably) is relative to the destination location, not to the link-dest location. This is probably clear from the man docs ... Commented Feb 27, 2021 at 11:00
  • Yes, that's right. It's fundamentally about copying from source to destination, and the link-dest is purely to help speed things on their way Commented Feb 27, 2021 at 13:01
0

This is sort of a workaround, as it uses Python. I have been experimenting with both diff and rsync most of the day. I simply cannot get rsync to report on files/directories which have been deleted in the source and are present in the link-dest location (e.g. in a dry run). diff does seem to do the job, but can be very lengthy with big directories, and there is no way for you to say "stop if you find a difference". The same applies to rsync for that matter.

import filecmp

def same_folders(dcmp):
    if dcmp.diff_files or dcmp.left_only or dcmp.right_only:
        return False
    for sub_dcmp in dcmp.subdirs.values():
        if not same_folders(sub_dcmp):
            return False
    return True

if same_folders(filecmp.dircmp( source_dir, link_dest_dir_path )):
    print( 'NO CHANGE' )
    # ... act accordingly

First experiments seem to show that this is a blindingly fast way to find a difference between two directories. Incidentally, I haven't yet looked into what sort of difference function is used to produce dircmp.diff_files (which lists files with the same paths but which differ).

left_only and right_only are about finding new files/directories in the source or ones which have been deleted from the source, respectively.

This obviously stops the second it finds a difference.

I know BASH much less than Python... I wonder if a BASH equivalent to the above is possible? Then it would be interesting to do a speed comparison...

1
  • Added example. I am trying to find whether there is a difference (any difference) between two directories (including their subtrees). If a difference is detected I would then do a proper (non-dry-run) rsync run. Commented Feb 26, 2021 at 7:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .