I have large directory that contains several millions+ sub-dir, each sub-dir contain 3 or less image files. What is the most efficient way for me to zip them so I can move it to a new server?

The sub-dir are 5 letter words compose of random letters + numbers.

For ex:
/foo/re63d/image.jpg, /foo/re63d/image2.jpg
/foo/4sf3g/image.jpg, /foo/4sf3g/image2.jpg
/foo/vj33s/image.jpg, /foo/vj33s/image2.jpg
and so on several millions times

I was thinking of using shell script to zip/tar them alphabetically. For ex every folders starting with

00* zip them into 0_0.zip
01* zip them into 0_1.zip
a0* into a_0.zip
a1* into a_1.zip
aa* into a_a.zip

This way it's easier for me to manage them. I want to be able to choose what sub section I want to zip at a time like aa* to af* as I want to be able control how long each process run. How would I go about doing this?

  • 4
    Not a real answer, but if what you want is to reliably move a lot of files to another server, I'd say rsync would be a better tool. – nxnev Dec 21 '18 at 23:34
  • I like the ability to see what I transfer. How much should I be worry about the integrity of the transfers if there's a broken connection? – Perz Dec 21 '18 at 23:43
  • 1
    @nxnev I would say it constitutes an answer because it's better for OPs use-case (It's not what he/she wants, but what he/she needs.) Perz: rsync is rock solid and will continue where it left off when there is a broken connection and has a compression algorithm built-in – Fabby Dec 21 '18 at 23:48
  • Thank you. I will have a look at rsync then. – Perz Dec 21 '18 at 23:52
  • What do you mean by efficient: Smallest size, least cpu time, least effort my you? – ctrl-alt-delor Dec 22 '18 at 12:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.