11

I want to take a 78gb folder and store it in a single file (for upload into a cloud service), as if I am compressing it in an archive, but I don't want any compression (I don't have that much CPU time available). Is there anyway that I can accomplish this, perhaps a terminal command I don't know about?

  • 1
    there is tar command in linux. tar -cf long.file.tar folder – Costas Apr 18 '15 at 20:33
  • 5
    FYI: There are fast compression programs, some which will be faster than your disk write speed, and many that will be faster than your network. Using one of those will actually speed things up, if the data is at all compressible. E.g., lzop is designed for speed. – derobert Apr 18 '15 at 20:57
20

Use tar: tar -cf my_big_folder.tar /my/big/folder

Restore the archive with tar -xf my_big_folder.tar -C /

-C will change to the root directory to restore your archive since the archive created above contains absolute paths.

EDIT: Due to the relatively big size of the archive, it'd be best to send it [directly] to its final location, using SSH or a mount point of the cloud resource/folder. For example, as Cole Johnson suggests :

tar -cf /network/mount/point/my_big_folder.tar /my/big/folder

or

tar -c /my/big/folder | ssh example.com "cat > my_big_folder.tar"

EDIT: As Blacklight Shining also suggests, If you want to avoid absolute paths, you can change to the big folder's parent and tar from there:

tar -cf /network/mount/point/my_big_folder.tar \
    -C /my/big/folder/location the_big_folder

or

tar -cC /my/big/folder/location the_big_folder | \
ssh example.com "cat > my_big_folder.tar"

Personal reflexions

Whether to include relative or absolute paths is a matter of personal preference.

There are cases absolute paths are obvious, e.g. for a restore in a disaster recovery situation. For local projects or collections it's common to archive a directory tree from the desired folder's parent so as to avoid cluttering the current directory, in case the archive is accidentally unpacked in-place. If big_folder lies somewhere deep in a standard *NIX hierarchy, it may make some sense to start archiving the first non-standard folder where big_folder deviates from and its directory tree from there.

Finally — going pedantic here — tar archive members are always relative since a) they may be restored in any directory and b) tar removes the leading / when creating an archive. I personally tend to always use -C when unpacking an archive.

  • To clarify the edit: do tar -cf /network/mount/point/folder.tar /folder instead of tarring to the local computer and then uploading if you can – Cole Johnson Apr 18 '15 at 21:46
  • Exact. The more I read my edit, the more I think it looks confused. Thanks for your insight. – user86969 Apr 18 '15 at 21:50
  • 1
    It's worth noting the way to avoid making the archive with absolute paths: tar -cf $archive.tar -C $source_dir .. If the destination machine doesn't have directory with the same path as the source directory, you'll need to do that. Just make sure to -C into a new directory when you extract (mkdir $destination_dir && tar -xf $archive.tar -C $destination_dir); else the archive contents will get dumped into whatever directory you happen to be in. – Blacklight Shining Apr 19 '15 at 22:39
2

Alternative: cpio

(cd /my/big/folder && find . -depth -print0 | cpio -0o > myfolder.cpio)

Unpacking to current directory:

cpio -id < myfolder.cpio

Caveats:

  • If use find /my/big/folder instead of cd, the archive will contain full paths and extraction will try to follow them;
  • Big files (> 2GB) may be a problem;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.