0

I want to copy or backup a large folder A to B without rsync (just because I can): https://stackoverflow.com/a/65485164/1707015.

It works but from time to time I want to clean up B and delete old files in B (when they are deleted in A).

What I have to get the deleted files in A (so I can then delete them in B also):

$ cat A_files.txt  # for example: think the small letters as paths like ./some/path/file.yaml
a
c
d
e
f

$ cat B_files.txt
a
b
c
d

$ \grep -f A_files.txt -F -v B_files.txt
b

(The backslash \ is just for not using any grep aliases with colors or something.)

This works, but only for small files. For files with more than 100 MB file names each, I need > 100 GB RAM :O

Does anyone have a more resource-efficient variant for me? Of course rsync, but that is not to be used here just for fun and for practice purposes.

3
  • I am confused: this question is not clear. Mainly with respect to file-names, file-content, lines. I can't work out what is what (without reading code) Jan 2 '21 at 18:12
  • Is it now a bit better?
    – uav
    Jan 2 '21 at 18:20
  • 1
    Yes I see it now. It has nothing to do with file-names: it could be any text. Jan 2 '21 at 19:45
3

For list conjunctions / subtraction, the standard command is comm. It works on lines of sorted files.

So for the lines of B_files.txt that are not in A_files.txt:

export LC_ALL=C # for a simple and deterministic order and allow any byte
                # in file names.
comm -23 <(sort A_files.txt) <(sort B_files.txt)

If the files are already sorted:

comm -23 A_files.txt B_files.txt

That approach (or yours) doesn't work for arbitrary file names though as file names can contain newline characters, so can't be represented by lines.

If you're on a GNU system, you can use NUL delimited records instead of lines and use the -z option to sort and comm.

Another approach is to use zsh's array conjunction/subtraction operators:

cd /path/to/A || exit
A_regular_files=(**/*(ND.))
cd /path/to/B || exit
B_regular_files=(**/*(ND.))

files_in_B_but_not_in_A=(${B_regular_files:|A_regular_files})

Also note that unless passed the -x option, grep does substring match. grep -F foo/bar matches on blah/foo/barrage for instance.

5
  • Thanks: comm -23 B_files.txt A_files.txt returns b. Woop-di-doo! Notice that I needed to change the order of the arguments. Otherwise it would output e and f.
    – uav
    Jan 2 '21 at 18:37
  • Dude. 41 seconds for all of my files! Without blowing the RAM! CPU load was first high - but who cares! Thanks!
    – uav
    Jan 2 '21 at 18:56
  • With -z: xargs: argument line too long :(
    – uav
    Jan 2 '21 at 19:46
  • @uav You need xargs -0 to work on nul-delimited records. Jan 2 '21 at 21:36
  • Of course, I did. Since I don't expect any line breaks in the paths, it must also work without -z or -0 - and it does.
    – uav
    Jan 2 '21 at 22:01
0

I came up with:

MY_SOURCE=A_files.txt
readarray -t MY_TARGET_ARRAY < B_files.txt
for LINE in "${MY_TARGET_ARRAY[@]}"; do
    if ! grep -q "${LINE}" "${MY_SOURCE}"; then
        echo "${LINE}";
    fi;
done

Not tested yet. Propably problems with missing -x and/or missing -F in grep.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.