2

I have a folder with 500,000 images, sorted in subfolders by year and month. I'd like to create a script that does this:

If the filename does not match any of the filenames in names.log then delete the file. names.log would contain filenames, like:

image1.jpg
photo3.jpg
redcar.jpg
balloon2323.jpg

etc... it has about 10,000 names of files I want to KEEP

I have PHP and Python on the server but I'm not sure what would be best for this. I haven't done any scripting before. Could anyone be so kind as to give me a snippet of code that would achieve that and let me know how to run it? Or maybe this can be achieved with a command instead?

  • Could you tell us what is the format of the file containing the filenames not to be deleted? Are they separater by coma, semicolon, colon, cartridge return, line feed? – dgsleeps Feb 22 '15 at 3:14
  • I haven't created it yet, I will essentially be exporting filenames of all attached images in my Wordpress. Let's assume they will be separated by comma. Incidentally, if you want to offer any suggestions for exporting those filenames from the WP database, I'm all ears. :) – location Feb 22 '15 at 6:09
3

This is fairly easy in Python with os.walk. Warning, untested code. I assume that the list of names contains one name per line with

#!/usr/bin/python2
import os
names_file = open('names.log')
names = set(line.rstrip('\n') for line in names_file.readlines())
names_file.close()
for root, dirs, files in os.walk('/path/to/top/directory'):
    for name in files:
        path = os.path.join(root, name)
        if os.path.isfile(path):
            if name not in names:
                print path
                #os.remove(path) # uncomment this line if you're happy with the set of files to remove
  • Thank you for the code, I got this when I ran it: Traceback (most recent call last): File "removefiles.py", line 3, in <module> names = set(names_file.readlines()) AttributeError: 'str' object has no attribute 'readlines' – location Feb 23 '15 at 1:26
  • I think I fixed it with names_file = open('names.log', 'r') now I get this error: Traceback (most recent call last): File "removefiles.py", line 8, in <module> path = os.path.join(root, files) File "/usr/lib64/python2.6/posixpath.py", line 65, in join if b.startswith('/'): AttributeError: 'list' object has no attribute 'startswith' – location Feb 23 '15 at 1:47
  • @location I fixed another typo – Gilles Feb 23 '15 at 8:59
  • Thanks, I tried again and it looks like we might be close. The script runs but it doesn't exclude the filenames that are in names.log, it just prints everything out. I tried having the full path and just the filenames in names.log, but it didn't work. Any thoughts? – location Feb 23 '15 at 19:42
  • @location I fixed one more error, and this time I tested the script, it should work. – Gilles Feb 24 '15 at 8:21
1
find -name '*.jpg' -print0 | grep -zZ -vf name.log | xargs -0 COMMAND

replace COMMAND by ls -l and if you like it by rm

Edit: The command presented is treating name.log as a set of regexps. @terdon remembered tha name.log is a list of filenames.

If filenames use the usual "normal" chars, this would probably be enough, but problems may occur if:

  1. Filenames / regexp include unusual chars like [, ] , etc. (in this case it may fail to deleted some files and even can delete some files whose name is in name.log!). To avoid this we can use grep -F or protect the special char in name.log.
  2. Regexp matches a substring of a filenames (in this case some files would not be deleted -- a.jpg would match all images ending with "a" like camera.jpg, banana.jpg).

For case 2, for the prefixes situation, -- we can add "/" in the beginning of the regexp.

sed 's!^!/!'  name.log > new.log
find -name '*.jpg' -print0 | grep -F -zZ -vf new.log | xargs -0 COMMAND

or even

find -name '*.jpg' -print0 | grep -zZFvf <(sed 's!^!/!' name.log) | xargs COMMAND

for case 2 , the suffix situation is less important because the image files have extensions. To solve this case properly, we need to say that "there is nothing after the filename": we need regexp, and special chars (example . [ ]) in filenames need to be protected.

sed -re 's!([].[])!\\\1!g; s!.*!/&$!' name.log > new.log
find -name '*.jpg' -print0 | grep -zZ -vf new.log | xargs -0 COMMAND
  • I rolled back your edit since, as far as I can tell, it only introduced a syntax error. You can't have the path as the last argument to find. – terdon Mar 7 '15 at 14:22
  • @terdon, thank you. In fact what I meant is find folder -name '*.jpg' -print0 |... but i prefer it simple (without "Folder"). – JJoao Mar 7 '15 at 16:31
  • Also, note that this will treat the file list as regexes, this means that if image1.jpg is in the file, you solution will not delete image1.jpg.bak either. It also means that foo.txt will also match fooAtxt. – terdon Mar 7 '15 at 16:36
  • @terdon, thank you for the bug report. I am not aloud to edit the answer today. Probably the number of "not delete" files will be very small... Anyway I would reduce it more with -F (-x is not possible because find adds ".../"). In order to make a more correct solution I would perl -nlE 's!([.])!/\\$1!g; say "/$_\$"' file.log > file.logre before, or ...|grep -f <(perl -nlE 's!([.])!/\\$1!g; say "/$_\$"' file.log) | ... – JJoao Mar 7 '15 at 20:23
0

This is really easy w/ pax. It has a notion of a -substitution option which can change filenames as they are written. You can specify more than one -substitution argument as well. And, most relevant here, is that selected members only have as many -substitution arguments applied as are necessary to successfully make one match, but any substitution which results in a null filename results in the matching file not being selected.

To demonstrate:

mkdir test; cd test
touch match nomatch
pax -ws '|^.*/match$|&|' -s '|.*||' ./ |
pax -v

The above makes and changes into a ./test directory, creates two files, then -writes a tar archive to a pipe w/ pax the contents of which a second pax -verbosely lists. The above prints:

-rw-r--r-- 1 mikeserv mikeserv 0 Feb 22 11:40 ./

...because ./match is matched before the final subtitution which substitutes away all characters in any file name.

And with pax you don't actually have to copy the contents of a file to its archive - you can use -rwl which is a copy operation that creates hardlinks.

So if your file was named paxscript and looked like...

cd -- "$1"
pax -rwvl \
    -s '|^.*/image1\.jpg$|&|' \
    -s '|^.*/photo3\.jpg$|&|' \
    -s '|^.*/redcar\.jpg$|&|' \
    -s '|^.*/balloon2323\.jpg$|&|' \
    -s '|.*||' ./ ../"${1##*/}.mirror"
cd - >/dev/null

...and then you ran it like...

. ./paxscript "$targetdir"

It would create a mirror of "$targetdir" in its parentdir containing only hardlinks to the filenames you wish to match. You could then verify the results are to your liking before doing rm -rf "$targetdir" and being rid of only all of the pathnames you do not want.


-1

If you will satisfy with bash I'd like to offer following algorithm (it could be realized on any scripting lanuage):

  1. Build list of present files: find /path_to_folder -name "*.jpg" -fprint files.tmp
  2. Sortfiles.tmp and name.log than compare its by comm -23 files.tmp name.log
  3. Pass the list of files unique for files.tmp to rm command

Be aware with path of files - in both files.tmp and name.log it could be same (full or relative to one folder). File name separator for name.log in the case would be newline.

  • 1
    You aren't quite there: the files are identified by names, not by their path, e.g. if names.log contains image1.jpg then /srv/2004/06/image1.jpg and /srv/2015/02/image1.jp are both supposed to be kept. – Gilles Feb 22 '15 at 15:45
  • 1
    @Gilles Why you play on behalf of OP? In any way there is no problem to use grep -vf name.log files.tmp instead of comm. – Costas Feb 22 '15 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.