Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to iterate over between 120k and 500k files. find handles this well.

find $PWD -type f -path "fragments/*.pdbqt"

For some reason I want to list the same set of files again, in the same order. I don't mind if the list is sorted alpha-numerically or by fstat calls or whatever. Using sort is not an option because it needs to know the full find output before proceeding, slowing down the script execution. Do you know some way to accomplish this?

Update. I described my problem poorly. While my script is processing this large number of files, a light outage might occur, and the UPS could go down. My script should continue from where it left off. How can I process the remaining files?

share|improve this question
2  
Save the output to a file and use the file the second time around. The order of find is not specified so therefore not something you can rely on. Or, redesign your script so you don't depend on the find order. –  jw013 Sep 19 '12 at 3:23
1  
You are right, and seems that I run into XY Problem. I'll update the question. –  Kanvuanza Sep 19 '12 at 3:37
    
Maybe record a result and/or timestamp for successfully completed files and skip the recently completed ones? I'd be interested to see what other ideas people can come up with. –  jw013 Sep 19 '12 at 4:31
    
@jw013 what you mean by using timestamps? Touching the processed files and find they by modified date? –  Kanvuanza Sep 19 '12 at 15:26
1  
Timestamp was just a vague suggestion because I don't know any of the specifics of what you are doing. Based on the comments I've seen, it sounds like what you could do is to generate the list of files first and then process that list. That way, if it gets interrupted, as long as you recorded where you were in the list (e.g. an offset), you can easily resume the operation at the file that was interrupted. –  jw013 Sep 19 '12 at 15:34

2 Answers 2

up vote 1 down vote accepted

The find command returns files in directory order. Directory order is not stable; in some filesystems, if you add or remove a file, it can affect the order of other files in the same directory.

I question your assertion that sort is not an option. Unless your processing of each file is extremely fast, building the list of files is pretty fast. Save it to a file so that it can be reused if the job is interrupted.

You'll need to know where you left off anyway, and resume from there. Resuming from an already-generated list of files will be easier than resuming some complex enumeration in the middle.

share|improve this answer
    
Yes, I did an assertion without verify. Sort and use a simple offset to resume is fine. –  Kanvuanza Sep 20 '12 at 11:59

Save the find output into a file and have your script use that file to as the list of files to process. As the script completes processing of a file, have it write the filename to another file. If the script is interrupted, it will be a simple matter to exclude the files already processed, e.g. with grep, when the script is restarted. Example:

touch files_already_processed
while IFS= read -r file
do
    # skip file if already processed
    grep -q -e "$file" files_already_processed && continue
    # process the file
    # processing code goes here
    echo "$file" >> files_already_processed
done <files_to_process
share|improve this answer
    
Your example scaled linearly up to 15k lines. files_already_processed with 1, 5, 10 and 15k lines took me 4, 22, 45 and 68s respectively. If this relation continues up to 250k it will take me ~3h just skipping already processed files. –  Kanvuanza Sep 19 '12 at 6:52
    
Obs: file_to_process was generated with seq 100000 and files_already_processed similarly but with varying sizes (as noted before). I stopped the code just after the files was skipped (ie. break on line 6). –  Kanvuanza Sep 19 '12 at 6:53
    
Comment out the grep for the first run since nothing has yet been processed. Worry about a more efficient implementation if the system really does crash with 250k lines in the file and you have to restart. Editing files_to_process and removing the first N lines would be a straightforward approach, with N = wc -l files_already_processed. –  Kyle Jones Sep 19 '12 at 6:57
    
Sure, it could crash with 250k lines. Remove the grep and the first N lines are good approaches, but my users aren't so smart. I'll try your sugestion tomorrow. Thank you. –  Kanvuanza Sep 19 '12 at 7:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.