Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

If I run tar -cvf on a directory of size 937MB to create an easily downloadable copy of a deeply nested folder structure, do I risk filling the disk given the following df -h output:

/dev/xvda1            7.9G  3.6G  4.3G  46% /
tmpfs                 298M     0  298M   0% /dev/shm

Related questions:

  • If the disk might fill up, why i.e. what will Linux (Amazon AMI) and/or tar be doing under the hood?
  • How can I accurately determine this information myself without asking again?
share|improve this question
    
I'm not sure if it's possible without processing the archive, but you can play around with --totals option. Either way if you fill the disk up you can simply delete the archive, imho. To check all options available you could go through tar --help. –  UVV Apr 10 at 8:02
3  
Tangentially: don't create the tarfile as root, a certain percentage of space on the disk is set aside for root exclusively, exactly for the kind of "I've filled the disk and now I can't login because that would write .bash_history or whatever" situation. –  Ulrich Schwarz Apr 10 at 9:01

4 Answers 4

up vote 16 down vote accepted

tar -c data_dir | wc -c without compression

or

tar -cz data_dir | wc -c with gzip compression

or

tar -cj data_dir | wc -c with bzip2 compression

will print the size of the archive that would be created in bytes, without writing to disk. You can then compare that to the amount of free space on your target device.

You can check the size of the data directory itself, in case an incorrect assumption was made about its size, with the following command:

du -h --max-depth=1 data_dir

As already answered, tar adds a header to each record in the archive and also rounds up the size of each record to a multiple of 512 bytes (by default). The end of an archive is marked by at least two consecutive zero-filled records. So it is always the case that you will have an uncompressed tar file larger than the files themselves, the number of files and how they align to 512 byte boundaries determines the extra space used.

Of course, filesystems themselves use block sizes that maybe bigger than an individual file's contents so be careful where you untar it, the filesystem may not be able to hold lots of small files even though it has free space greater than the tar size!

https://en.wikipedia.org/wiki/Tar_(computing)#Format_details

share|improve this answer
    
Thanks Jamie! What is '- mysql' doing here? Is that your filename? –  codecowboy Apr 10 at 8:50
    
Just changed that... it is the path to your data directory. –  FantasticJamieBurns Apr 10 at 8:50
    
Not that it really matters, but using the argument combination -f - to tar is redundant, since you can simply leave out the -f argument altogether to write the result to stdout (i.e. tar -c data_dir). –  OrbWeaver Apr 10 at 14:10

tar itself can report on the size of its archives with the --test option:

tar -cf - ./* | tar --totals -tvf -

The above command writes nothing to disk and has the added benefit of listing the individual filesizes of each file contained in the tarball. Adding the various z/j/xz operands to either side of the |pipe will handle compression as you will.

OUTPUT:

...
-rwxr-xr-x mikeserv/mikeserv         8 2014-03-13 20:58 ./somefile.sh
-rwxr-xr-x mikeserv/mikeserv        62 2014-03-13 20:53 ./somefile.txt
-rw-r--r-- mikeserv/mikeserv       574 2014-02-19 16:57 ./squash.sh
-rwxr-xr-x mikeserv/mikeserv        35 2014-01-28 17:25 ./ssh.shortcut
-rw-r--r-- mikeserv/mikeserv        51 2014-01-04 08:43 ./tab1.link
-rw-r--r-- mikeserv/mikeserv         0 2014-03-16 05:40 ./tee
-rw-r--r-- mikeserv/mikeserv         0 2014-04-08 10:00 ./typescript
-rw-r--r-- mikeserv/mikeserv       159 2014-02-26 18:32 ./vlc_out.sh
Total bytes read: 4300943360 (4.1GiB, 475MiB/s)

Not entirely sure of your purpose, but if it is to download the tarball, this might be more to the point:

ssh you@host 'tar -cf - ./* | cat' | cat >./path/to/saved/local/tarball.tar

Or to simply copy with tar:

ssh you@host 'tar -cf - ./* | cat' | tar -C/path/to/download/tree/destination -vxf -
share|improve this answer
    
The reason I am doing this is that I believe the directory in question has caused the output of df -i to reach 99%. I want to keep a copy of the directory for further analysis but want to clear the space –  codecowboy Apr 10 at 8:19
    
@codecowboy In that case, you should definitely do something like the above first. It will tar then copy the tree to your local disk in a stream without saving anything to the remote disk at all, after which you can delete it from the remote host and restore it later. You should probably add -z for compression as goldilocks points out, to save on bandwidth mid-transfer. –  mikeserv Apr 10 at 8:24
    
@TAFKA'goldilocks' No, because it's 99% of inodes, not 99% of space. –  Gilles Apr 10 at 21:56
    
-i right, sorry! –  goldilocks Apr 11 at 12:29
    
@mikeserv your opening line mentions the --test option but you then don't seem to use it in your command which immediately follows (it uses --totals) –  codecowboy Apr 30 at 10:46

-cvf does not include any compression, so doing that on a ~1 GB folder will result in a ~1 GB tar file (Flub's answer has more details about the additional size in the tar file, but note even if there are 10,000 files this is only 5 MB). Since you have 4+ GB free, no you will not fill the partition.

an easily downloadable copy

Most people would consider "easier" synonymous with "smaller" in terms of downloading, so you should use some compression here. bzip2 should now-a-days be available on any system w/ tar, I think, so including j in your switches is probably the best choice. z (gzip) is perhaps even more common, and there are other (less ubiquitous) possibilities with more squash.

If you mean, does tar use additional disk space temporarily in performing the task, I am pretty sure it does not for a few reasons, one being it dates back to a time when tape drives were a form of primary storage, and two being it has had decades to evolve (and I am certain it is not necessary to use temporary intermediate space, even if compression is involved).

share|improve this answer

The size of your tar file will be 937MB plus the size of the metadata needed for each file or directory (512 bytes per object), and padding added to align files to a 512-byte boundary.

A very rough calculation tells us that another copy of your data will leave you with 3.4GB free. In 3.4GB we have room for about 7 million metadata records, assuming no padding, or fewer if you assume an average of 256 bytes' padding per file. So if you have millions of files and directories to tar, you might run into problems.

You could mitigate the problem by

  • compressing on the fly by using the z or j options to tar
  • doing the tar as a normal user so that the reserved space on the / partition won't be touched if you run out of space.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.