-2

I am currently using 2 scripts. 1 script to filter the files by year. 2nd script into which sits inside the year subfolders to filter by month # (01-12).

Is there something that would be better than what I have below?

All of the files are in 1 directory:

SOURCE: ./tape_backup/sync1/* (1.4 million files) TARGET: ./tape_backup/<1990 - 2019>/<01-12>/ (Organized by year/month)

The syntax of the filename: A1000_T195_R256393_D120498094600

_D = Unimporant

D = Date start

1-2 = month

3-4 = Day (unimportant)

5-6 = Year

7-* = Timestamp (unimportant)

Hence why I do the whole D????98

Something that would read like this:

for f in ./sync1 do
(check the month and year, then mv it into the month and years folder /)
done

I'm on Cygwin and the server 2012 r2

#C:/cygwin/bin/bash
for filename in ./* ; do
 if [[ $filename == *D??90 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1990/

 elif [[ $filename == *D??91 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1991/

 elif [[ $filename == *D??92 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1992/

 elif [[ $filename == *D??93 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1993/

 elif [[ $filename == *D??94 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1994/

 elif [[ $filename == *D??95 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1995/

 elif [[ $filename == *D??96 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1996/

 elif [[ $filename == *D??97 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1997/

 elif [[ $filename == *D??98 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1998/

 elif [[ $filename == *D??99 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/1999/

 elif [[ $filename == *D??00 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2000/

 elif [[ $filename == *D??01 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2001/

 elif [[ $filename == *D??02 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2002/

 elif [[ $filename == *D??03 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2003/

 elif [[ $filename == *D??04 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2004/

 elif [[ $filename == *D??05 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2005/

 elif [[ $filename == *D??06 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2006/

 elif [[ $filename == *D??07 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2007/

 elif [[ $filename == *D??08 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2008/

 elif [[ $filename == *D??09 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2009/

 elif [[ $filename == *D??10 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2010/

 elif [[ $filename == *D??11 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2011/

 elif [[ $filename == *D??12 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2012/

 elif [[ $filename == *D??13 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2013/

 elif [[ $filename == *D??14 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2014/

 elif [[ $filename == *D??15 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2015/

 elif [[ $filename == *D??16 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2016/

 elif [[ $filename == *D??17 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2017/

 elif [[ $filename == *D??18 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2018/

 elif [[ $filename == *D??19 ]] ; then
      mv $filename /cygdrive/d/RAID5/RAID200/invoices/1/2019/
 fi
done
4

If you're going to be calling mv 1.4 million times, it's going to be slow - no way around that. But you already know the pattern of the filenames, so why not batch them, calling multiple files with a single mv:

for yy in {1990..2019}
do
    y=${yy#[0-9][0-9]}  # remove first two digits of year
    mv ./*D??"$y" "/cygdrive/d/RAID5/RAID200/invoices/1/$yy"
done

Or, if this runs into "argument length exceeded" errors, use xargs:

for yy in {1990..2019}
do
   y=${yy#[0-9][0-9]}
   printf "%s\0" ./*D??"$y" |   # NUL-delimited filenames for xargs -0
       xargs -0 -r mv -t "/cygdrive/d/RAID5/RAID200/invoices/1/$yy"
done
  • I updated my OP - hope it makes more sense... THANK YOU! – Mike D Jul 16 at 16:26
  • I got your first one to work. With some modification, perfectly. – Mike D Jul 17 at 1:31
5

Doing 1.4 million calls to mv to move things between different drives will be slow. Try calling mv fewer times.

Assuming that the ?? in your pattern is supposed to match the century:

for year in {1990..2019}; do
    find . -maxdepth 1 -type f -name "*D$year" \
        -exec mv -t "/cygdrive/d/RAID5/RAID200/invoices/1/$year/" {} +
done

This would loop over all relevant years. For each year, it will execute a find command that would move as many files that match the given pattern as possible at once, in batches.

The code assumes that you use GNU mv (for the -t option) and GNU find (or any find that has -maxdepth) together with bash. If your source directory does not contain any subdirectories, then you may remove -maxdepth 1 from the command.

  • I updated my OP - I hoep it makes more sense! Thank you!!! – Mike D Jul 16 at 16:26
0

Just so everyone knows - I used this option, by muru. I actually tried both options posted above.

With a slight modification... I was doing about 3 files a second with my original 50 lines of code. Now I'm doing about 30 - 50 a second.

for yy in {1998..2018} do y=${yy#[0-9][0-9]} # remove first two digits of year mv -v ./sync1/_D????$y "/cygdrive/d/RAID5/RAID200/tape_backup/$yy" #mv -v ./sync1/_D????'$y' "/cygdrive/d/RAID5/RAID200/tape_backup/$yy" done

:)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.