Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have about 1 million files in this directory: /home/username/images/

Each of the files are called something like: 012345678910(Place)_0_20120414185957_28841.jpg with the timestamp part of the filename changing on each picture.

The code below contains code to sort/move the files into this date structure: /home/username/sorted/2012/04/14/18/name_of_file.jpg

For a small sample of files it works fine, but for the huge directory my putty terminal gets disconnected after outputting

Directory $newdir does not exist.  Creating same.

I had other code which always died with the error code argument list too long.

Here is the code:

#!/bin/bash
ALLFILES=(images/*)
for ((i=0; i<${#ALLFILES[*]}; i+=30000));
do
    set $(echo "${ALLFILES[@]:i:30000}" | awk -F_ '{print $1, $2, $3, $4, $5}')
    fullyear=$3
    year=$(echo $fullyear |cut -c1-4)
    month=$(echo $fullyear |cut -c5-6)
    day=$(echo $fullyear |cut -c7-8)
    hour=$(echo $fullyear |cut -c9-10)
    newdir=$(echo /home/username/sorted/$year/$month/$day/$hour/)
    if ! [ -d $newdir ]; then
        echo Directory $newdir does not exist.  Creating same.
        mkdir -p $newdir;
    fi
    mv "${ALLFILES[@]:i:30000}" $newdir;
done

Any ideas why the connection will not hold while performing the large loop?

share|improve this question
    
Have you tried turning on xtrace (set -x at the beginning or invoke bash with -x) for debugging? –  jw013 Apr 17 '12 at 10:52
add comment

2 Answers 2

up vote 4 down vote accepted

Try to run it in screen session. Or even try another construction. I believe find + sed will work better then pure bash:

find images/ -name "*.jpg" | sed 's%^[^_]*_[^_]*_\([0-9][0-9][0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\).*%mkdir -p "/home/username/sorted/\1/\2/\3/\4" \&\& mv "&" "/home/username/sorted/\1/\2/\3/\4/"%'

This is just to show, how sed make commands to execute. Adding e after last % will force command executing:

find images/ -name "*.jpg" | sed 's%^[^_]*_[^_]*_\([0-9][0-9][0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\).*%mkdir -p "/home/username/sorted/\1/\2/\3/\4" \&\& mv "&" "/home/username/sorted/\1/\2/\3/\4/"%e'

ps. You don't need to use in bash

day=$(echo $fullyear |cut -c7-8)

Bash can do it itself without echo | cut :

day=${fullyear:6:2}
share|improve this answer
    
I never knew about the sed command. The snippet you gave works perfectly ... well it's been working away for the last few mins generating the folders and moving the files in correct order. This was my first time ever creating a *nix script, I'm coming from Windows. The server I'm connecting to is a web-hosting provider, not mine so I can only use Putty to connect. –  PadraigD Apr 17 '12 at 13:13
    
I also didn't know that Bash can do the segmenting of the variables. That's good to know. Thanks very much for your help Rush. –  PadraigD Apr 17 '12 at 13:19
add comment

I'm also sorting images into date-structured directories, but I have a slightly different approach. I want my images to go to their respective YYYY-MM directories, based on their timestamps. So what I do is that I start by ls -l *.jpg > tmp.txt the image folder, then I feed this tmp.txt into a loop to get the timestamp for each file. I haven't found a way to get the timestamp otherwise.

Here is my code:

#!/bin/bash
hostdir="/home/Photos/"
destdir="/tmp/sorted"

cd $hostdir

touch /tmp/tmpsort.txt
ls -l *.jpg > /tmp/tmpsort.txt

while read line
do
    filename=$(echo $line | awk '{print $8}')
    filedate=$(echo $line | awk '{print $6}')
    filedir=${filedate:0:7}

    if [ ! -d $destdir/$filedir ]; then
        mkdir -p $destdir/$filedir
    fi

    # Let's skip files that were already sorted from a previous run
    if [ ! -f $destdir/$filedir/$fiename ]; then
        cp $filename $destdir/$filedir/
    fi

done < /tmp/tmpsort.txt
rm /tmp/tmpsort.txt

I don't have millions of images to sort and if I did, this code would take quite some time to execute. But it works as intended.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.