I have CCTV server that is backed up daily to a remote ftp server using an lftp command. The CCTV server saves videos to a new folder each day, the backup runs once a day at 1am, so each backup only affects 2 folders. After 28 days the local copies are all deleted.

The command I currently use is:

mirror --reverse --use-cache --allow-chown --allow-suid --no-umask --verbose

The file transfer rate when moving files has been consistent for 2 years, but the wait between folders is slowly increasing. This means that while the file transfer rate is 1Mb/s if you take the wall clock time that backing up 2GB takes each day, the average speed after 2 years is now down to 0.5Mb/s.

Is anything in my command causing the process to bloat? --use-cache for example?

Could I have the mirror command to run one thread per folder so that it can get on with loading folder contents while it's uploading other files?

  • How many directories already exist on the destination? If the directory containing those is very large due to a very large amount of subdirectories, then creating new subdirectories can become very slow. – wurtel Sep 4 '18 at 12:40
  • The destination directory structure is /cctv/%year%/%month%/%day%/files.mp4 so each directory only contains a few subdirectories (up to 31) but the current year will have a total of around 300 subdirectories including recursion. And /cctv/ contains 600 in the same way. Is cumulative bad or just at each level? – Martin KS Sep 4 '18 at 12:44
  • 1
    Directory size matters per directory. Has there been rotation of these files, i.e. old versions have been removed? Depending on what filesystem is in use that can lead to filesystem fragmentation. – wurtel Sep 4 '18 at 12:53
  • nope, but it's a shared server so other users may have done so. I'll ask. – Martin KS Sep 4 '18 at 13:04

After trying clearing the cache, and removing the --use-cache command entirely, my service provider admitted they were having "network issues" their end, which was causing ls commands to run very slowly. Fortunately detailed logs from my end were able to demonstrate their error, and now show things running at full speed.

Keep your logs people!!

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.