2

We have a SAN with a lot (over 5 million) of files. Because of some freak mistake from a developer script, some files have been deleted and we now have to figure out which files got deleted.

Now, I do have a list with the filenames (from our database) that need to be checked, which looks like this:

49
50
51
52

I have figured out how I can use the find command to do this for single and multiple files, as follows:

find /mnt/SAN/documents/ -type f -name"92441_1"

This works, but I cannot use this command with over 200K+ files to check. So I am wondering if anyone knows a way to get the find command to take input from my file, or if there is another command I can use to get this done.

2

I'd do something like (assuming GNU tools):

find /mnt/SAN/documents -type f -print0 | awk -F / '
  NR == FNR{check[$0]; next}
  $NF in check {print "found:", $0; delete check[$NF]}
  END {
    for (i in check)
      print "Not found:", i
  }' filename.list RS='\0' -

Which would find one occurrence for each filename in filename.list.

Or to report all occurrences:

find /mnt/SAN/documents -type f -print0 | awk -F / '
  NR == FNR{check[$0]; notfound[$0]; next}
  $NF in check {print "found:", $0; delete notfound[$NF]}
  END {
    for (i in notfound)
      print "Not found:", i
  }' filename.list RS='\0' -
  • i have tried using this in a testing enviroment with a couple of numbers of files that i know are there, but somehow they are all returning to Not found: [number], so it is taking input from the filename.list. i am however not sure what i am doing wrong here. any idea? – Sefiris Jun 2 '16 at 13:56
1

Use something like

find /mnt/SAN/documents/ -type f | perl -ple 's,^.*/,,' > files_currently_present

to generate a list of files currently on disc, without paths and then use

comm -2 -3 filelist_from database files_currently_present

to compare it to the list from your backup and generate a list of mssing files.

0

The straightforward way is to read the file names from your file using a shell loop and then run multiple find commands in the background:

while IFS= read -r file; do
    find /mnt/SAN/documents/ -type f -name "$file" &
done < fileList.txt > foundFiles.txt

That, however, will launch 200K+ instances of find and will probably bring your machine to its knees. A better approach would be to build a complex find command, giving it each file name:

$ printf 'find /mnt/SAN/documents/ -type f '; while IFS= read -r file; do printf -- '-name "%s" -o ' "$file"; done < fileList.txt | sed 's/-o $/\n/'
find /mnt/SAN/documents/ -type f -name "49" -o -name "50" -o -name "51" -o -name "52" 

You can then run the command itself either by copy/pasting it or with:

eval $(printf 'find /mnt/SAN/documents/ -type f '; \
    while IFS= read -r file; do 
        printf -- '-name "%s" -o ' "$file"; done < fileList.txt | 
            sed 's/-o $/\n/')

However, this will also break if you have too many files, so you'll want to run it in batches:

for i in $(seq 1 100 $(wc -l < fileList.txt)); do 
    k=$((i+100)); 
    printf 'find /mnt/SAN/documents/ -type f '; 
    sed -n "$i,${k}p" fileList.txt | 
    while IFS= read -r file; do 
        printf -- '-name "%s" -o ' "$file"; 
    done  | sed 's/-o $/\n/';   
done

That will create separate find commands for each batch of 100 files in your list which you can execute with eval as shown above, or just save in a file and run the file:

for i in $(seq 1 100 $(wc -l < fileList.txt)); do 
    k=$((i+100)); 
    printf 'find /mnt/SAN/documents/ -type f '; 
    sed -n "$i,${k}p" fileList.txt | 
    while IFS= read -r file; do 
        printf -- '-name "%s" -o ' "$file"; 
    done  | sed 's/-o $/\n/';   
done > script.sh && bash script.sh > foundFiles.txt

Note that Stéphane's approach, starting with existing files and checking which are missing would almost certainly be better here (unless the existing files are more than the missing ones). In a similar vein, you could first build a list of all existing files and then use comm to compare that to the list of target files (since you say you have a list of files, I will assume that your file names will never contain newline characters):

find /mnt/SAN/documents/ -type f | sort > found
comm -13 <(sort found) <(sort fileList.txt)

The comm command will print all lines that are in fileList.txt but not in found.

-1

It is very time consuming to run find against every file name in 200K+files. If I were you , I would run find ${FILESROOT} > /tmp/SANfiles then

for filename in $(cat my_database_files)
do
  grep "${filename}" /tmp/SANfiles > /dev/null; r=${?}
  if [ ${r} -eq 0 ]
  then
    echo ${filename} >> /tmp/existing_files
  else
    echo ${filename} >> /tmp/missing_files
  fi
done

depending on how the file names in your database is formatted, you might want to modify the variable name in the for loop, but I think you've got the gist of my idea.

  • Never iterate over a list of file names with a for loop (commonly known as Bash Pitfall #1). This will break if any of the file names contains a space or a globbing character. – terdon May 31 '16 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.