1

I have some 200k files in a folder, i want to organize them based on time of birth(creation time). i wrote the below script but it is way too slow. i want to improve it. how can i do it?

#!/usr/bin/env bash
echo Input directory is $1
input_directory=$1

## Find those files that are older than a month

inputfiles=$(hadoop fs -ls  $input_directory | sed '1d;s/  */ /g' | cut -d\  -f8)

for filename in $inputfiles
  do
   echo processing $filename

   hadoop fs -test -d  $filename

   lastcommand=$?

   if [ "$lastcommand" == "1" ];then

        year=$(date -d "`hadoop fs -stat $filename`" +%Y)
        month=$(date -d "`hadoop fs -stat $filename`" +%m)

        hadoop fs -test -d $input_directory/$year-$month
        lastcommand2=$?

        [[ "$lastcommand2" == "1" ]] && hadoop fs -mkdir -p $input_directory/$year-$month;

        hadoop fs -mv $filename $input_directory/$year-$month/
    else
        echo not a file
   fi
  done

closed as off-topic by Rui F Ribeiro, jimmij, RalfFriedl, andcoz, Romeo Ninov Sep 12 '18 at 8:07

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question has been posted on multiple sites. Cross-posting is strongly discouraged; see the help center and community FAQ for more information." – Rui F Ribeiro, andcoz, Romeo Ninov