Filter phots into directories by date range

Question

I have set up our family mobile phones to automatically backup all photos taken to a NAS (Qnap TS-231P), one folder per mobile. For one phone, for some reason a lot of photos were copied multiple times without warning. In addition, it seems some photos were taken in 48MP mode, so are large files. There are 314281 files totalling 2.19 TB.

Now the folder is so large that it never loads and can't be practically browsed/used (in Windows, in the Qnap Software, on the apps on my phone, etc). I have used SSH to access the Linux operating system of the NAS, and can navigate to the appropriate directory. I tried du -h on the directory but 40 Minutes later it's still thinking (the size above comes from the Qnap software).

I have a feeling that there is no real problem with the files, so having the files in multiple folders might help.

How can I enter a command to filter the files by modified date into e.g. one folder per month or perhaps one folder per quarter year?

For bonus points, would it be possible to get (i.e. export to a .txt or .csv) a report on the 314281 files before I do anything?

I'm quite a newbie at Linux, so bear with me.

Answer 1

I have something similar

I have been using a shellscript for a similar purpose since April 2019, so it is pretty well tested.

• It is hardcoded for the mountpoint of my data partition. You must modify that.

• It is moving file names to lower case. You may want to skip that.

• It is not optimized for the huge amount of files that you have. Maybe it is too slow and needs tweaks to make it faster, maybe even to use parallel processing.

• The find command and commands around it in 'main' will only waste time for you. I think you should remove it, at least for the huge initial batch job. (This could be the report you ask for, but I think it will only waste time.)

• It is interactive if a file with the same name that is different is found where to move a file. This could probably be removed to increase speed, at least for the huge initial batch job.

• Please remember that the mv command is fast when moving within the same file system but slow (copying and deleting) between file systems (between partitions).

After modifications, please test it on a small number of files before starting the big job.

pict2dirtree:

#!/bin/bash


function doit {

 fildir=$(find -name "$1" -printf "%TY/%Tm/%Td")
 lcas=${1##.*}
 lcas=${lcas,,}
 fildir="/media/multimed-2/Photos/$fildir"
 mkdir -p "$fildir"
 filnam="$fildir/${lcas}"
#echo "fildir=$fildir"
#echo "filnam=$filnam"
#echo "$lcas"
#exit
 if test -s "$filnam"
 then
  diff -q "$1" "$filnam"
  if [ $? -eq 0 ]
  then
   rm "$1"
  else
   read -p "move to '_$lcas' (y/N) " ans
   if [ "$ans" == "y" ]
   then
    mv -i "$1" "${fildir}/_$lcas"
   fi
  fi
 else
  mv "$1" "$filnam"
 fi
}

########################################################################

# main

########################################################################

if [ "$(whoami)" == "root" ]
then
 echo "Usage: Do *not* run with sudo"
 exit
fi

find -maxdepth 1 -type f -exec ls -l --time-style=long-iso "{}" \;|tr -s ' ' ' '|cut -f 6,8|sort|more
echo "Move files from the current directory (no subdirectories)"
read -p "Are you ready to move these files (y/N) " ans
if [ "$ans" != "y" ]
then
 exit
fi

for i in *
do
 if test -f "$i"
 then
  doit "$i"
 else
  echo "skipping $i"
 fi
done