I have roughly 1 million images on a directory. The files were numbered from 1 to n. I am using for loop to iterate over each image. Since each iteration is checked by individuals, only certain number of iterations could be done in a day. When I begin the loop again the subsequent day, the loop obviously begins from the first file again.

I saved the files iterated through the loop in a text file and read the last line of the text file before the loop starts every time. I am trying to use the last read file as a beginning for the for loop.

The following is the code done so far:

fil=$( tail -n 1 readfiles.txt )

for f in $query
      python ~/runprog.py --query $f

I am not sure how to use the $fil as my starting point in the for loop and the start iterating subsequent files from thereon.

  • You could, once you processed an image, move this image to a different directory, say to_be_reviewed. Dec 14 '17 at 11:04
  • @user1934428 Thank you...I tried this with a small set of images...and it worked...but since the size of images were big and also were in large numbers....I wanted to keep all the images in just one file.
    – Apricot
    Dec 14 '17 at 11:18
  • You all pictures in the same file? For sure you don't. You have them in the same folder. BTW, moving the file to a different folder doesn't take longer for large pictures than for small ones, as long as both folders are in the same file system. Dec 15 '17 at 6:04

If your readfiles.txt contains all already processed files, you can use grep to look up if a certain was done or not.

After running the python script, update that file with the processed file.

for f in /ImageFolder/*.jpg; do
   if ! grep -q "$f" readfiles.txt; then 
      python ~/runprog.py --query "$f"
      echo "$f" >> readfiles.txt
  • excellent...thank you. Works like a charm. I was struggling and stuck with using the last file.
    – Apricot
    Dec 14 '17 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.