3

I have hundreds of directories under /var/Recording, and each directory there will have subdirectories, including files, hard links and soft links.

I want to compress all directories under /var/Recording to create a single compressed file.

Which command would give me the best compression? tar or cpio (especially considering the fact that I have hard and soft link files).

Also, what is the right syntax of the tar/cpio command ?

  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1034
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1033
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1038
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1037
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1036
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1041
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1040
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1039
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1044
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1043
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1042
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1047
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1046
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1045
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1049
  drwxr-x--- 2 root root 4096 Nov 22 18:57 458ca4e8-0edf-4204-9f9b-9c3dc02953c5.1048
  .
  .
  .
  .
  .
3
  • 1
    The "best" compression will depend on what exactly you are compressing. Do you actually want to compress or just create an archive. tar (and I think cpio) do not actually compress by default. Also, if you have sound files, these will not be compressed much whatever you do. You would need to convert them first. So, please edit and clarify what you need and also explain what the ls output you show is. Is that /var/Recording? – terdon Nov 23 '14 at 11:39
  • I want to compress all directories to one file ( in order to decrease usage of file to one file ) – maihabunash Nov 23 '14 at 11:54
5

cpio (the older of the two utilities counting shipping with UNIX) only used to have hard link support for the -p option (i.e. copying from filesystem to filesystem), but the newc output format (not the default one cpio uses) also supports hard links in the output file. (GNU) tar supports hard links without any special options. A comparison can be found here.

So if you run a test with a large hard linked file and 100 small files:

$ mkdir tmp
$ dd if=/dev/urandom of=tmp/blabla bs=1k count=1024
1024+0 records in
1024+0 records out
1048576 bytes (1,0 MB) copied, 0,0764345 s, 13,7 MB/s
$ ln tmp/blabla tmp/hardlink
$ tar cvf tmp.tar tmp
$ find tmp -print0 | cpio -0o > out.cpio
4104 blocks
$ find tmp -print0 | cpio -0o --format=newc > outnewc.cpio
2074 blocks
$ xz -9k out.tar outnewc.cpio
$ bzip2 -9k out.tar outnewc.cpio
$ ls -l out*
-rw-rw-r-- 1 anthon users 2101248 Nov 23 12:30 out.cpio
-rw-rw-r-- 1 anthon users 1061888 Nov 23 12:30 outnewc.cpio
-rw-rw-r-- 1 anthon users 1055935 Nov 23 12:30 outnewc.cpio.bz2
-rw-rw-r-- 1 anthon users 1050652 Nov 23 12:30 outnewc.cpio.xz
-rw-rw-r-- 1 anthon users 1157120 Nov 23 12:30 out.tar
-rw-rw-r-- 1 anthon users 1055402 Nov 23 12:30 out.tar.bz2
-rw-rw-r-- 1 anthon users 1050928 Nov 23 12:30 out.tar.xz

You see that the uncompressed versions (outnewc.cpio and out.tar) give cpio an advantage and that compressing them with xz -9 gives better results than bzip2 -9 (gzip is usually much worse than either). And that compression with xz minimizes the tar and cpio output difference. Compression is however heavily dependent on the data, and also on the ordering of the data in the archives, so you should really test this on (a sample of) your real data.

If you want to compress in parallel, you might want to look at my article here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.