Searching folders with patterns listed in a CSV file and copy them to another location

Question

I have thousands of folders with files inside them and I want to copy some of them in another directory. I have a .csv file with two columns with part of the folder name (the folder contain one string value or another, not both).

• Example of folder names:

  PLASMA_32150129_B5/
  PLASMA_AAA3891784_B3/
  ...
  

• The CSV file has no header and the fields are separated by ,:

  32150129,AAA0616938
  32140203,AAA3891784
  32140204,AAA0617237
  32140205,AAA0617261
  32140206,AAA0617285
  ...
  

• I found this little script as a starting point:

  while IFS=, read -r file rest
  do
    find /path/to/Main_directory -type d -name "${file}" -exec cp '{}' /path/to/New_directory/ \;
  done < mylist.csv
  

Now I need to specify that

• the csv values are just a pattern (like *_32150129 _*), and
• I want to try pattern in the first column first, and if that doesn't generate a match, try with the other one.

Is this possible?

Thank you!

Answer 1

Assuming that only one of the columns can actually match, an "or"-type approach might be possible. To do so, perform just a minor modification to your script:

while IFS=, read -r pattern1 pattern2
do
   find /path/to/start -type d \( -name "*_${pattern1}_*" -o -name "*_${pattern2}_*" \) -exec cp -r '{}' /path/to/target \;
done < filelist.csv

• The -o operator will ensure that either of the two name patterns generate a match. As long as not both can match (see prerequisite), this will be equivalent to "the second, if the first is not present"
• The parentheses (escaped to prevent interpretation by the shell) are to ensure correct operator precedence.
• Placing the shell arguments to -name in double-quotes will ensure the * remains literally there without being glob-expanded (necessary so that find performs the pattern matching during the search, not the shell before passing it to find), while still allowing the shell variables ${pattern1} and ${pattern2} to be expanded.
• The -r option to cp is necessary if you want to copy directories and their contents.

Answer 2

The simplest, crudest approach which should work if you only ever have one match for each pattern in your file, is to just let the shell do the work:

tr , '\n' < mylist.csv  | 
    while read pat; do cp -r source_dir/*"$pat"* target_dir/ 2>/dev/null; done

That will first convert all , in file.csv to newline characters:

$ tr , '\n' < file.csv
32150129
AAA0616938
32140203
AAA3891784
32140204
AAA0617237
32140205
AAA0617261
32140206
AAA0617285

We then read each pattern into the variable $pat and just blindly copy source_dir/*$pat* (i.e. anything in source_dir/ matching $pat) to the target dir. If nothing matches then nothing is copied, and we get an error message which we discard by using 2>/dev/null.

---

for a more sophisticated approach, you could do something like this:

## make sure we don't match the glob pattern itself when no matching
## directories are found
shopt -s nullglob

mainTargetDir=/path/to/Main_directory
sourceDir=/path/to/start
while IFS=, read -r f1 f2; do 
    sourceFiles=("$sourceDir"/*"$f1"* "$sourceDir"/*"$f2"*)
    [ -z "${sourceFiles[@]}" ] || 
        cp -r -- "${sourceFiles[@]}" $mainTargetDir 
done < file.csv