Detecting "bad" images from the command line

Question

I've restored a huge number of images from accidental formatting of a disk.

A great number of these images are corrupted, ie. 100% green or 70% grey. Unfortunately, they all still have EXIF data so I can't offhand use that to sort out which images are bad.

Answer 1

Using Imagemagick

This one gives you the total number of pixels in the image:

identify -verbose  ~/www/pictures/ISTI-F.jpg | sed -n '/.*Pixels: /s///p'

and this one gives you the dominant colour name and number of pixels:

convert ~/www/pictures/ISTI-F.jpg -format "%c" histogram:info: | sort -nr -t: | head -1 | sed 's/\(.*\):.*#\(......\).*/\2\1/'

Example:

$ identify -verbose ISTI-F.jpg | sed -n '/.*Pixels: /s///p'
1920000
$ convert ISTI-F.jpg -format "%c" histogram:info: | sort -nr -t: | head -1 | sed 's/\(.*\):.*#\(......\).*/\2\1/'
FFFFFF   1667711

This can be the basis for writing a shell script, but it is not good at checking many files, as Imagemagick is incredibly flexible, but slow

Using Octave

The following Octave script can be called directly from the command line. Its arguments should be two directory names errdir and baddir and a list of image files. Files for which the LibMagick++ library used by Octave gives a warning or error are moved to errdir; files for which the last 25% of rows is of the same colour are moved to baddir; other files are left untouched. A terse report is given on standard output.

If you want just the report, without moving files, don't give directory names as first two arguments.

#!/usr/bin/octave -qf

threshold = 0.25;

usage = "Usage is: badfiles <file...> OR badfiles <errdir> <baddir> <file...>\n";
files\n";
assert(nargin>0, usage);
dryrun = isfile(argv{1});
if !dryrun
  errdir = argv{1};
  baddir = argv{2};
  assert(isfolder(errdir) && isfolder(baddir) && isfile(argv{3}), usage);
endif

start = 1 + 2*(!dryrun);
for fname = argv()(start:end)'
  q = [];
  f = fname{};

  warning error
  try
    q = imread(fname{});
  catch err
  end_try_catch
  warning on

  if isempty(q)
    printf("error\t");
    dryrun || movefile(f, errdir);
  else
    qt = all(q == q(end,1,:) ,2);
    qtt = squeeze(all(qt, 3));
    r = 1 - find(qtt==0, 1, 'last') / size(q, 1);
    if (r > threshold)
      printf("bad--%02d\t", ceil(100*r));
      dryrun || movefile(f, baddir);
    else
      printf("good-%02d\t", ceil(100*r));
    endif
  endif

  disp(f);
endfor

You need to have Octave installed for this to work. Copy the above in a file called badimage, make it executable with chmod +x badimage and test the script like ./badimage *.jpg: you will see a list of files with their status (good, bad, error).

Look at the output and possibly change the threshold inside the script for more aggressive (lower threshold) or more conservative (higher threshold) behaviour. You can test it as long as you want, as it does not move or change files if you only give image file names as arguments.

Once you are satisfied with the results, create two directories mkdir errpics badpics. Then call the script as ./badimage errpics badpics *.jpg. This way the script moves your files in the directories as described above.

CAVEAT: after using this script, be sure to check a good sample of images that are marked as bad or errored before deleting them!

The algorithm that detects bad images is reliable for photos, but not necessarily for drawings, logos, diagrams, graphs, which may contain wide legitimate areas of the same colour. The errored files may be bad or damaged from LibMagick++'s point of view, but be well readable on the usually more lenient image visualisers.