Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have about 1 million files in this directory: /home/username/images/

Each of the files are called something like: 012345678910(Place)_0_20120414185957_28841.jpg with the timestamp part of the filename changing on each picture.

The code below contains code to sort/move the files into this date structure: /home/username/sorted/2012/04/14/18/name_of_file.jpg

For a small sample of files it works fine, but for the huge directory my putty terminal gets disconnected after outputting

Directory $newdir does not exist.  Creating same.

I had other code which always died with the error code argument list too long.

Here is the code:

#!/bin/bash
ALLFILES=(images/*)
for ((i=0; i<${#ALLFILES[*]}; i+=30000));
do
    set $(echo "${ALLFILES[@]:i:30000}" | awk -F_ '{print $1, $2, $3, $4, $5}')
    fullyear=$3
    year=$(echo $fullyear |cut -c1-4)
    month=$(echo $fullyear |cut -c5-6)
    day=$(echo $fullyear |cut -c7-8)
    hour=$(echo $fullyear |cut -c9-10)
    newdir=$(echo /home/username/sorted/$year/$month/$day/$hour/)
    if ! [ -d $newdir ]; then
        echo Directory $newdir does not exist.  Creating same.
        mkdir -p $newdir;
    fi
    mv "${ALLFILES[@]:i:30000}" $newdir;
done

Any ideas why the connection will not hold while performing the large loop?

share|improve this question
    
Have you tried turning on xtrace (set -x at the beginning or invoke bash with -x) for debugging? –  jw013 Apr 17 '12 at 10:52
add comment

1 Answer

up vote 4 down vote accepted

Try to run it in screen session. Or even try another construction. I believe find + sed will work better then pure bash:

find images/ -name "*.jpg" | sed 's%^[^_]*_[^_]*_\([0-9][0-9][0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\).*%mkdir -p "/home/username/sorted/\1/\2/\3/\4" \&\& mv "&" "/home/username/sorted/\1/\2/\3/\4/"%'

This is just to show, how sed make commands to execute. Adding e after last % will force command executing:

find images/ -name "*.jpg" | sed 's%^[^_]*_[^_]*_\([0-9][0-9][0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\).*%mkdir -p "/home/username/sorted/\1/\2/\3/\4" \&\& mv "&" "/home/username/sorted/\1/\2/\3/\4/"%e'

ps. You don't need to use in bash

day=$(echo $fullyear |cut -c7-8)

Bash can do it itself without echo | cut :

day=${fullyear:6:2}
share|improve this answer
    
I never knew about the sed command. The snippet you gave works perfectly ... well it's been working away for the last few mins generating the folders and moving the files in correct order. This was my first time ever creating a *nix script, I'm coming from Windows. The server I'm connecting to is a web-hosting provider, not mine so I can only use Putty to connect. –  PadraigD Apr 17 '12 at 13:13
    
I also didn't know that Bash can do the segmenting of the variables. That's good to know. Thanks very much for your help Rush. –  PadraigD Apr 17 '12 at 13:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.