3

I've restored a huge number of images from accidental formatting of a disk.

A great number of these images are corrupted, ie. 100% green or 70% grey. Unfortunately, they all still have EXIF data so I can't offhand use that to sort out which images are bad.

Example bad image

3
  • 5
    Just a tip: note that if you look into your file with hexdump it is filled with nulls at the end. That's the missing data. You could write a program or script calculatig relation between actual non-null data and 0x00 0x00 0x00 0x00 in seqence. This relation could be than treated as a measure of chance that the file has been corrupted. Commented Apr 10, 2020 at 12:28
  • 2
    Imagemagick has some tooling (identify -verbose filename), but can you spell out what "corrupted" might mean in more detail? Would a picture of a field that was 90% "green" be OK or not OK, for example?
    – Jeff Schaller
    Commented Apr 10, 2020 at 14:11
  • 1
    @JeffSchaller, In this case, I'd probably set up something like: up to 25% OK; up to 40%, put aside for human validation; above that, just trash (or archive).
    – Jan
    Commented Apr 11, 2020 at 7:11

1 Answer 1

6

Using Imagemagick

This one gives you the total number of pixels in the image:

identify -verbose  ~/www/pictures/ISTI-F.jpg | sed -n '/.*Pixels: /s///p'

and this one gives you the dominant colour name and number of pixels:

convert ~/www/pictures/ISTI-F.jpg -format "%c" histogram:info: | sort -nr -t: | head -1 | sed 's/\(.*\):.*#\(......\).*/\2\1/'

Example:

$ identify -verbose ISTI-F.jpg | sed -n '/.*Pixels: /s///p'
1920000
$ convert ISTI-F.jpg -format "%c" histogram:info: | sort -nr -t: | head -1 | sed 's/\(.*\):.*#\(......\).*/\2\1/'
FFFFFF   1667711

This can be the basis for writing a shell script, but it is not good at checking many files, as Imagemagick is incredibly flexible, but slow

Using Octave

The following Octave script can be called directly from the command line. Its arguments should be two directory names errdir and baddir and a list of image files. Files for which the LibMagick++ library used by Octave gives a warning or error are moved to errdir; files for which the last 25% of rows is of the same colour are moved to baddir; other files are left untouched. A terse report is given on standard output.

If you want just the report, without moving files, don't give directory names as first two arguments.

#!/usr/bin/octave -qf

threshold = 0.25;

usage = "Usage is: badfiles <file...> OR badfiles <errdir> <baddir> <file...>\n";
files\n";
assert(nargin>0, usage);
dryrun = isfile(argv{1});
if !dryrun
  errdir = argv{1};
  baddir = argv{2};
  assert(isfolder(errdir) && isfolder(baddir) && isfile(argv{3}), usage);
endif

start = 1 + 2*(!dryrun);
for fname = argv()(start:end)'
  q = [];
  f = fname{};

  warning error
  try
    q = imread(fname{});
  catch err
  end_try_catch
  warning on

  if isempty(q)
    printf("error\t");
    dryrun || movefile(f, errdir);
  else
    qt = all(q == q(end,1,:) ,2);
    qtt = squeeze(all(qt, 3));
    r = 1 - find(qtt==0, 1, 'last') / size(q, 1);
    if (r > threshold)
      printf("bad--%02d\t", ceil(100*r));
      dryrun || movefile(f, baddir);
    else
      printf("good-%02d\t", ceil(100*r));
    endif
  endif

  disp(f);
endfor

You need to have Octave installed for this to work. Copy the above in a file called badimage, make it executable with chmod +x badimage and test the script like ./badimage *.jpg: you will see a list of files with their status (good, bad, error).

Look at the output and possibly change the threshold inside the script for more aggressive (lower threshold) or more conservative (higher threshold) behaviour. You can test it as long as you want, as it does not move or change files if you only give image file names as arguments.

Once you are satisfied with the results, create two directories mkdir errpics badpics. Then call the script as ./badimage errpics badpics *.jpg. This way the script moves your files in the directories as described above.

CAVEAT: after using this script, be sure to check a good sample of images that are marked as bad or errored before deleting them!

The algorithm that detects bad images is reliable for photos, but not necessarily for drawings, logos, diagrams, graphs, which may contain wide legitimate areas of the same colour. The errored files may be bad or damaged from LibMagick++'s point of view, but be well readable on the usually more lenient image visualisers.

4
  • 1
    Thanks for the suggestion! This could be made to a script that checks the ratio between the two numbers fairly trivially. Unfortunately, at almost 10s per image for convert, this would take quite a lot of time given 116.000 images... Even assuming most pictures is less than 10s, it's more than a week of 100% compute time.
    – Jan
    Commented Apr 11, 2020 at 7:09
  • 1
    Thanks @pot, but it throws an error: pastebin.com/raw/Vh4LLFBh . I'll try looking into it when my kids are asleep. Maybe it's pertinent that I'm running macOS with the homebrew version of octave. GNU Octave, version 5.2.0, Octave was configured for "x86_64-apple-darwin19.4.0".
    – Jan
    Commented Apr 12, 2020 at 12:42
  • I actually cleaned this directory of files that ImageMagick didn't want to play with, but Octave seems more fidgety. Does it have a way of catching errors? Maybe just define a directory to move "faulty" files to and another for "cleaned" files.
    – Jan
    Commented Apr 12, 2020 at 16:39
  • I think I owe you a beer. Not just any beer. A good, Danish beer!
    – Jan
    Commented Apr 13, 2020 at 8:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .