3

I'm trying to compare a folder with 1400 subfolders to one with a 1.399. I need to know which subfolder is missing.

I tried this:

diff -rq dir1/ dir2/ | grep dir1/ | awk '{print $4}' > difference1.txt

But it's been like 6 hours yet and no output. ls in the folders is nearly instant so is there a faster approach than diff?

2

This is a fairly simple task for rsync.

rsync -n -av dir1/ dir2/ | grep '/$' | grep -Fvx './'

This does a dry run of rsyncing dir1's contents into dir2, and displays only the lines ending in a slash (the directories), except possibly for the top-level directory itself.

Here's a simple test/demo:

$ # create 1500 folders in dir1:
$ for i in $(jot -w %04d 1500); do mkdir -p dir1/dir-$i; done
$ # clone dir1 to dir2, then remove one directory:
$ rsync -a dir1/ dir2/
$ rmdir dir2/dir-0749/
$ # rsync -n will tell you which one is missing:
$ time rsync -n -av dir1/ dir2/ | grep '/$' | grep -Fvx './'
dir-0749/

real    0m0.038s
user    0m0.001s
sys     0m0.041s

If you would like to adapt this technique to also check for folders missing from dir1 but present in dir2, use rsync's --delete option:

$ rmdir dir1/dir-0479/
$ time rsync -nav --delete dir1/ dir2/ | grep '/$' | grep -Fvx './'
deleting dir-0479/
dir-0749/

The deleting ... line tells you that dir-0479 is absent from dir1 but present in dir2.

1

Try either ls dir1 > file1; ls dir2 > file2; diff file1 file2

or if you have vimdiff and tree which is the most useful way with sub directories, you could use

tree dir1 > file1; tree dir2 > fiel2; vimdiff file1 file2

Though, running ls is much faster than tree which is what you are after.

If the difference is in a sub-directory in dir1 eg.

dir1
 - dir2
 - dir3
    - dir4
vs

dir2
 - dir2
 - dir3

You would need to do

ls -R dir1 > file1; ls -R dir2 > file2; vimdiff file1 file2

  • This would try to compare names of non-directories too, of which we know nothing about. – Kusalananda Jul 18 at 9:51
1
diff -u <(ls dir1) <(ls dir2)

This will make sure diff does not look inside the subdirs.

Or try this if you are brave ;)

diff -u <(find dir1/ -maxdepth 1 -type d -exec basename {} \;) <(find dir2/ -maxdepth 1 -type d -exec basename {} \;)
  • diff -u <(cd dir1; find -type d | sort) <(cd dir2; find -type d | sort) ? – Hannu Jul 8 at 15:51
  • 1
    =) ............ – Hannu Jul 8 at 15:51
  • comm -23 <(ls /dir1 |sort) <(ls /dir2 |sort) also worked for me and was very fast too – Freedo Jul 8 at 16:19
0

[EDIT] After posting I noticed @freedo already mentioned comm before me. [/EDIT]

First store lists of the directory trees into files:

# Using parenthesis ensures the command are executed inside a subshell, so
# changing directory (`cd`) back afterwards is not needed.

(cd dir1 ; find -type d | sort >/tmp/list1)
(cd dir2 ; find -type d | sort >/tmp/list2)

Then this command will show what's present in dir1 but not in dir2:

comm -23 /tmp/list1 /tmp/list2

While this commmand will show what is in dir2 but not in dir1:

comm -13 /tmp/list1 /tmp/list2

And this will show what is present in both:

comm -12 /tmp/list1 /tmp/list2

Finally, this will show all information above at once, in 3 columns:

comm /tmp/list1 /tmp/list2

The comm utility seems not be very well known. But it's probably installed by default on most if not all Linux systems. In Debian it's in the "coreutils" package, which is marked "essential".

Little known command, but comm proved very useful to me once, while I was solving nasty problems in a big filesystem (> 25 Tb). I feel I owe it to comm to advocate its use.

-1

diff <(tree -d dir_A) <(tree -d dir_B)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.