0

I've searched high and low but it seems all but one (Find and list duplicate directories) topic I've found actually deals with my situation, and the result isn't quite what I need.

Edit: Here's some example data to help show what I'm trying to accomplish. Below is a list of two sets of directories.

idx1
idx1/defaultdb
idx1/defaultdb/thaweddb
idx1/defaultdb/colddb
idx1/defaultdb/db
idx1/defaultdb/db/rb_1558019513_1558019454_4_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx1/defaultdb/db/rb_1558019513_1558019454_4_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx1/defaultdb/db/rb_1541720372_1541194569_2_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx1/defaultdb/db/rb_1541720372_1541194569_2_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx1/defaultdb/db/rb_1558019538_1558019538_5_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx1/defaultdb/db/rb_1558019538_1558019538_5_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx1/defaultdb/db/db_1558019449_1558019418_3_9542F466-F8CA-49EB-8120-5409B813F147
idx1/defaultdb/db/db_1558019449_1558019418_3_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx1/defaultdb/db/rb_1558019389_1558018342_3_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx1/defaultdb/db/rb_1558019389_1558018342_3_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx1/defaultdb/db/rb_1558019898_1558019898_7_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx1/defaultdb/db/rb_1558019898_1558019898_7_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx1/defaultdb/db/db_1557947113_1557947083_0_9542F466-F8CA-49EB-8120-5409B813F147
idx1/defaultdb/db/db_1557947113_1557947083_0_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx1/defaultdb/db/rb_1549909440_1549908720_1_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx1/defaultdb/db/rb_1549909440_1549908720_1_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx1/defaultdb/db/test
idx1/defaultdb/db/rb_1558019813_1558019569_6_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx1/defaultdb/db/rb_1558019813_1558019569_6_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx1/defaultdb/db/rb_1558020652_1558020018_8_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx1/defaultdb/db/rb_1558020652_1558020018_8_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx1/defaultdb/db/db_1541720372_1541194569_2_9542F466-F8CA-49EB-8120-5409B813F147
idx1/defaultdb/db/db_1541720372_1541194569_2_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx1/defaultdb/db/GlobalMetaData
idx1/defaultdb/db/db_1558019873_1558019567_4_9542F466-F8CA-49EB-8120-5409B813F147
idx1/defaultdb/db/db_1558019873_1558019567_4_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx1/defaultdb/db/db_1558020619_1558019927_5_9542F466-F8CA-49EB-8120-5409B813F147
idx1/defaultdb/db/db_1558020619_1558019927_5_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx1/defaultdb/db/rb_1557960001_1557771284_0_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx1/defaultdb/db/rb_1557960001_1557771284_0_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx1/defaultdb/db/db_1558032446_1558018050_1_9542F466-F8CA-49EB-8120-5409B813F147
idx1/defaultdb/db/db_1558032446_1558018050_1_9542F466-F8CA-49EB-8120-5409B813F147/rawdata

idx2
idx2/defaultdb
idx2/defaultdb/thaweddb
idx2/defaultdb/colddb
idx2/defaultdb/db
idx2/defaultdb/db/db_1558019813_1558019569_6_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx2/defaultdb/db/db_1558019813_1558019569_6_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx2/defaultdb/db/rb_1557947113_1557947083_0_9542F466-F8CA-49EB-8120-5409B813F147
idx2/defaultdb/db/rb_1557947113_1557947083_0_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx2/defaultdb/db/db_1558019513_1558019454_4_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx2/defaultdb/db/db_1558019513_1558019454_4_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx2/defaultdb/db/rb_1558019449_1558019418_3_9542F466-F8CA-49EB-8120-5409B813F147
idx2/defaultdb/db/rb_1558019449_1558019418_3_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx2/defaultdb/db/db_1558019898_1558019898_7_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx2/defaultdb/db/db_1558019898_1558019898_7_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx2/defaultdb/db/db_1558019538_1558019538_5_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx2/defaultdb/db/db_1558019538_1558019538_5_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx2/defaultdb/db/rb_1541720372_1541194569_2_9542F466-F8CA-49EB-8120-5409B813F147
idx2/defaultdb/db/rb_1541720372_1541194569_2_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx2/defaultdb/db/db_1541720372_1541194569_2_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx2/defaultdb/db/db_1541720372_1541194569_2_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx2/defaultdb/db/test
idx2/defaultdb/db/rb_1558032446_1558018050_1_9542F466-F8CA-49EB-8120-5409B813F147
idx2/defaultdb/db/rb_1558032446_1558018050_1_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx2/defaultdb/db/db_1557960001_1557771284_0_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx2/defaultdb/db/db_1557960001_1557771284_0_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx2/defaultdb/db/db_1558019389_1558018342_3_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx2/defaultdb/db/db_1558019389_1558018342_3_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx2/defaultdb/db/GlobalMetaData
idx2/defaultdb/db/5_9542F466-F8CA-49EB-8120-5409B813F147
idx2/defaultdb/db/5_9542F466-F8CA-49EB-8120-5409B813F147/rawdata
idx2/defaultdb/db/db_1549909440_1549908720_1_AB8C9371-027D-4FE0-B2F3-BAF93F106480
idx2/defaultdb/db/db_1549909440_1549908720_1_AB8C9371-027D-4FE0-B2F3-BAF93F106480/rawdata
idx2/defaultdb/db/rb_1558019873_1558019567_4_9542F466-F8CA-49EB-8120-5409B813F147
idx2/defaultdb/db/rb_1558019873_1558019567_4_9542F466-F8CA-49EB-8120-5409B813F147/rawdata

Say I have the following one:

idx1/defaultdb/db/rb_1558019513_1558019454_4_AB8C9371-027D-4FE0-B2F3-BAF93F106480

I want to check in the idx2 directory to see if defaultdb/db/rb_1558019513_1558019454_4_AB8C9371-027D-4FE0-B2F3-BAF93F106480 exists in it, and if it does I want to print it.

The ultimate goal is for each every complete directory (directory has no sub directories, I don't want defaultdb showing up but rather the children) to be unique across all top level directories being a list of sub-directories that exist in the two different top level directories. From there I will delete one of them.

Edit2: This is what my current working copy looks like. Might be some bugs I need to fix. It accepts to directories in the current path, finds identical path names and removes the ones in the second directory.

#!/bin/bash
echo 'Chosen directories must reside in the current directory.'
echo 'This will find duplicate sub directories between the two and delete the ones in the second path.'
echo ''
read -p 'First directory to compare:' DIR1
read -p 'second directory to compare:' DIR2

depth="${DIR1//[^\/]}"
depth="${#depth}"
recurse='..'

for ((i=1; i<=depth; i++)) {
    recurse="${recurse}/.."
}

cd $DIR1; find . -type d > "$recurse"/list.txt; cd "$recurse"
cd $DIR2; find . -type d >> "$recurse"/list.txt; cd "$recurse"
echo 'Paths found:'
echo ''
awk 'seen[$1]++ {print $1}' list.txt | grep -v "db$" | grep -v "\.$"
echo ''
read -p 'Delete paths in ${DIR2}? (y/n)' bool
case 'y' in
    $bool)
    echo 'deleting:'
    awk 'seen[$1]++ {print $1}' list.txt | grep -v "db$" | grep -v "\.$"    
    cd $DIR2
    awk 'seen[$1]++ {print $1}' "$recurse"/list.txt | grep -v "db$" | grep -v "\.$" | xargs rmdir   
    echo ''
esac
Improve this question
3
  • 1
    (1) Please define more clearly what you want.  Examples would help. (2) Hint: ls is probably a very bad tool for this.
    – Scott - Слава Україні
    May 16, 2019 at 17:42
  • What do you mean by "complete path"? Do you want a list of files whose relative path is identical in the two trees?
    – bitinerant
    May 16, 2019 at 17:50
  • I've reworded it with more detail. @bitinerant no, files are explicitly irrelevant. The sub-directories of two different directories must have the same path (/a1/foo/bar/ and a2/foo/bar/ would be matches because the sub directories are the same, but /a1/foo /and /a2/foo/ are NOT matches because they have sub directories
    – user2752794
    May 16, 2019 at 19:27

1 Answer 1

Reset to default
1

I'd go with something like this :

  1. list directories on the idx1 side : cd idx1/defaultdb; find . -type d > path/to/list.txt

  2. list directories on the idx2 side : cd idx2/defaultdb; find . -type d >> path/to/list.txt

  3. find duplicates : awk 'seen[$1]++ {print $1}' path/to/list.txt

NB :

  • this is just an overall concept. Still needs polish to become a script ;-)
  • both find commands will have to actually write into the same file. Choose its path accordingly.
Improve this answer
3
  • If you do find idx1/defaultdb, the paths will all begin with idx1/defaultdb, so no duplicates will be found
    – muru
    May 17, 2019 at 7:16
  • @muru Exact. I've edited my answer.
    – Httqm
    May 17, 2019 at 7:46
  • very clever, I'll give that a shot. I didn't consider dumping them into a file for some reason.
    – user2752794
    May 17, 2019 at 18:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .