Delete all files in directories except those whose path are listed in a file

Question

Given the following files:

data/A/a.txt
data/B/b.pdf
...
date/P/whatever.log
...
data/Z/z.jpg

I would like to delete all files in the data/A/, data/B/, ..., data/Z/ directories except those files that are situated under one of the directories listed in the file data/dont_clean.txt. For example, if we have data/P listed in data/dont_clean.txt then nothing should be touched under data/P/, etc.

Something like:

find data/ -mindepth 2 -maxdepth 2 -type f -not -path {listed in data/dont_clean} -delete

Of course it is not a valid command.

I have also tried variants of

find data/ -mindepth 2 -maxdepth 2 -type f -exec grep data/dont_clean.txt '{}' \;

but I only created either an invalid command or I had no idea why I got the output I did.

I am using bash on Ubuntu 12.10

Answer 1

This is code that I only roughly tested but might layout an approach for you to take. Assuming you have a file, ignore.txt like this:

1/
2/

Sample data

And I had sample directories with files in them like this:

$ mkdir -p dirs/{1..5}
$ touch dirs/{1..5}/afile

Resulting in this:

$ tree dirs/
dirs/
|-- 1
|   `-- afile
|-- 2
|   `-- afile
|-- 3
|   `-- afile
|-- 4
|   `-- afile
`-- 5
    `-- afile

Example run

Now if we run this command against this tree:

$ find dirs/ -type f -print0 | fgrep -zFvf ./ignore.txt
dirs/5/afiledirs/4/afiledirs/3/afile

We can see that we're only getting back the files that are in directories not listed in ignore.txt.

So we can add a rm to the end to remove the non-excluded files.

$ find dirs/ -type f -print0 | fgrep -zFvf ./ignore.txt | xargs -0 rm -f

Checking we can see that it worked:

$ tree dirs/
dirs/
|-- 1
|   `-- afile
|-- 2
|   `-- afile
|-- 3
|-- 4
`-- 5

Problems to be worked out

One big problem with this approach is that the strings in the ignore.txt file might match other portions of the directory structure. So some care needs to be paid to making sure that the strings in this file are unique in the way that you expect.

Some blocking could be put around the strings so that they're anchored to the beginning or the end of the string to protect them.

Details

The above commands are doing the following:

1. finding all the files under the directory dirs
2. filtering out any files that are under a directory present in the igonre.txt file
3. passing the filter list via xargs to the rm -f command

Answer 2

Sounds like a case for the comm command.

list of files to not delete in "keeper"
ls >current
comm -23 current ../keeper | more

check that it's the correct list

comm -23 current ../keeper | xargs rm

Answer 3

xargs and find combination

Demo:

My Files :

[root@mail tmp]# find data/ -type f
data/A/d.txt
data/A/b.txt
data/A/a.txt
data/A/c.txt
data/B/e.txt
data/B/g.txt
data/B/f.txt
data/B/i.txt

Exclude List

[root@mail tmp]# cat exclude 
data/A/a.txt
data/B/e.txt

find with xargs

[root@mail tmp]# find data/ -type f $( xargs  -I{} echo -n " -not -path {} " < exclude )
data/A/d.txt
data/A/b.txt
data/A/c.txt
data/B/g.txt
data/B/f.txt
data/B/i.txt

Seems Output is OK, let delete but before that make sure output of following command is OK, then you can remove second last echo command.

[root@mail tmp]# find data/ -type f $( xargs  -I{} echo -n " -not -path {} " < exclude ) | xargs -n1 echo rm -rf
rm -rf data/A/d.txt
rm -rf data/A/b.txt
rm -rf data/A/c.txt
rm -rf data/B/g.txt
rm -rf data/B/f.txt
rm -rf data/B/i.txt

Answer 4

The only thing that needs to solved is to create a program that:

• reads a line from the stdout,

• then echos this line if it doesn't start with any of the strings listed in the data/dont_clean.txt file.

I couldn't solve this with a bash script easily so I wrote a little C++ program called my_program that does exactly this. Then I get what I want if I run:

find data/ -mindepth 2 -maxdepth 2 -type f | my_program | xargs rm -f 

However, I leave this question still open, I am still interested in a pure shellscript solution.

Answer 5

You could do it in two steps: mark (to make them distinctive) and then remove unprotected.

If all these files are from the same user, then you can use the list to change ownership or group to another user. Then use find to remove the rest and then change the user back. If all the files have the same read permissions (so you know how to switch back), you can for instance just remove read permission (to flag them) because it's much easier than changing users (no need for root privileges):

while read file; do chmod u-w -R "$file"; done < "data/dont_clean.txt"
find data/ -mindepth 2 -maxdepth 2 -type f -writable -delete
while read file; do chmod u+w -R "$file"; done < "data/dont_clean.txt"

Edited (added recursion) to protect whole directories. Changed from read to write permissions to avoid problems with chmod recursion (it couldn't read directory it just changed).

Answer 6

You could process data/dont_clean.txt into a list of globs to be ignored by bash via GLOBIGNORE e.g. if dont_clean.txt contains directory paths like

data/P
data/some_dir

the result would be GLOBIGNORE=data/P/*:data/some_dir/*;
you would then just remove all regular files that the glob data/*/* expands to:

GLOBIGNORE=$(sed -n 's|$|/*|;H;1h;$x;s|\n|:|gp' data/dont_clean.txt)
for f in data/*/*; do [[ -f "$f" ]] && rm "$f"; done
unset GLOBIGNORE

The above assumes sane directory names in your list. Use printf %s\\n instead of rm if you want to see which files will be removed.