3

I have about 1 million small files, that take up about 20 GB. I need to upload them to a host that only has 23 GB free space. I would prefer compressing the files locally to increase the transmission speed (would it work btw?), but I don't think I will be have enough free space to unpack the file (there is no need to keep the archive). Is there any compression format that would allow decompressing the file with that little free space?

  • 2
    Just using rsync would probably be enough (even without compression unless the network is really slow). Creating an archive and then unpacking it would take longer, probably even if you somehow created it through a pipe directly to the remote system and read from the pipe with tar -x (all files would have to be read and written regardless). – Kusalananda Feb 7 at 10:30
  • 2
    You can ask rsync to use compression over the networ, or if it uses ssh you can ask ssh to use compression. – meuh Feb 7 at 10:52
  • 1
    @meuh, so rsync -avz should do the trick? – Fluffy Feb 7 at 10:59
  • 1
    stream it, Don't save it: that is un-compress as it arrives, but don't save it. tar supports reading from standard in. – ctrl-alt-delor Feb 7 at 11:36
  • 2
    Yes, rsync -avz will show at the end ...speedup is 4.5 or some number which shows by what factor the data was reduced. But try it on a subset of files first, because if you have very small files rsync compresses each file, and this may not be as good as tar czf - fromdir | ssh remote 'tar xzf - -C todir', where tar does the compression of the stream. – meuh Feb 7 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.