sliding window of JPGs from thousands of files

Question

My machine has a camera attached that can take thousands of jpgs a day. I am looking for a way of remotely looking at the last 30 to 60 mins of footage via a cloud storage site such as box.com.

I thought that copying the last 5 mins of files to a mounted remote drive (davfs2) every min and then deleting files each 10 mins that were older than an hour would be a good solution; but it has caused big problems! It caused my machine not to be able to connect via SSH; thus requiring me to power it off.

Now, even if I try and delete the files, it is still re-copying them to the mounted drive. I've had to un-mount the drive, but don't seem to be able to clear the davfs2 cache.

Is there a fundamental problem with my approach?

I put this in my crontab:

*/1 * * * * sudo find /mnt/ -type f -cmin -5 -exec cp -pn '{}' /home/pi/box/street_pictures/ \;
*/10 * * * *  sudo find /home/pi/box/street_pictures -cmin +60 -type f -exec rm '{}' \;

/nmt is the folder with the thousands of pictures. /home/pi/box/street_pictures is my mounted drive.

Answer 1

Why sudo? If it runs as root anyway, put it in the root crontab?

Scanning a (remote) directory with thousands of files in it can take a long time. If it takes longer than a minute, you could have multiple cron jobs running concurrently, fighting one another for resources and many remote filesystems aren't able to handle concurrent accesses well.

To avoid concurrent cron jobs, you could use some kind of locking mechanism, or since it will run non-stop anyway you could put it in a service script instead:

while [ 1 ]
do
    sleep 60 &
    cp jpg stuff
    wait # for sleep in case 60 seconds have not yet passed
done

Maybe you could rename already copied files to a ignored subdir?

-exec {} \; also isn't very efficient (starts a new cp process for each file). Try -exec cp --target-directory=/home/pi/... {} + instead, if your programs support such an option.

If the filenames are in any way predictable (contain the date or incrementing number), it may be more efficient to rely on them rather than their timestamp, to avoid unnecessary / time consuming stat() of file properties.