copy huge number of files into date structured directory order

Question

I need to copy around 40.0000 files into date structured folders.
example file:

/var/public/voicelogging/quality_monitoring/20151209/bbbbbb_I_20151209-185841_xxxxxx_12434_89343.WAV

Is one of the many files I need to copy to /home/username/logging/

The file name has 2 variables in it that I need to use:

bbbbbb_I_20151209-185841_xxxxxx_12434_89343.WAV

20151209 is of course the date
12434 is the id of the user who made the file.

What I need is a script/one liner that can search in a dir for the user id.

Then create a dir with the user id in /home/username/logging. After it created the folder it needs to create a dir for every date it can find.
And place every file in to the right userid/date directory.
example of result dir.

/home/username/logging/12434/20151209/bbbbbb_I_20151209-185841_xxxxxx_12434_89343.WAV

I have build a one-liner for making the date dir's , but I still need to make the user id dir myself.

find /var/public/voicelogging/quality_monitoring/ -type f -name "*12434*" | sed -r 's/^.{65}//' | cut -c1-8 | xargs -I {} mkdir {} /home/username/logging/12434

How can I copy the right file to the right place?

Answer 1

One way with find and install:

find /var/public/voicelogging/quality_monitoring -name \*.WAV -exec sh -c '
bn=${0##*/}; x=${bn%%-*}; dt=${x##*_}; y=${bn%_*}; id=${y##*_} 
install -D "$0" "/home/username/logging/${id}/${dt}/${bn}"' {} \;

this uses parameter expansion to extract the date: ${dt} and the user id: ${id} from the filename and then uses install to copy each file to the corresponding userID/date directory (this is because I'm lazy) - without install replace the last line with:

dest=/home/username/logging/${id}/${dt}; mkdir -p "${dest}" && cp "$0" "${dest}"' {} \;

If you prefer to loop over those "date" directories and then loop over the .WAV files in each dir:

for d in /var/public/voicelogging/quality_monitoring/*; do
  dt=${d##*/}
  for f in $d/*.WAV; do
    bn=${f##*/}; y=${bn%_*}; id=${y##*_}
    dest=/home/username/logging/${id}/${dt}
    mkdir -p "${dest}" && cp "${f}" "${dest}"
  done
done

---

If you have zsh it's easier and shorter with zmv (also because zsh is smarter and you can nest variable expansions e.g. ${${file%_*}##*_} would be enough to extract the user ID):

dtcp () {                                                                     
mkdir -p $3 && cp $1 $2 $3
}
autoload zmv
zmv -n -p dtcp '/var/public/voicelogging/quality_monitoring/(*)/(*).WAV' \
'/home/username/logging/${${2%_*}##*_}/$1'

The (*)s create back references that can be used in the second parameter as $1, $2 etc.
Here zmv with -p executes the function dtcp instead of mv. The function creates the directory and then copies the file to the newly created directory. The arguments (not to be mistaken for the back references above) are:
$1 : --
which means end of options
$2 : /var/public/voicelogging/quality_monitoring/(*)/(*).WAV'
that is the file that has to be copied and
$3 : /home/username/logging/${${2%_*}##*_}/$1
which is the destination
Note that -n stands for dry-run; remove it to actually run the command.