Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

This question did not help me (despite having the same title). So I post this even though this is a duplicate question.

As far as I can tell, the total in du -k includes all the subdirectories and indicates I have 77 megabytes of data

/raid/fpuData/oldOutput>du -ks
77063332        .
/raid/fpuData/oldOutput>tar -cvzf ../oldOutput.tar.zip *

The backup is sill running, but already the file is considerably bigger than 77 megabytes

/raid/fpuData>ls oldOutput.tar.zip
-rw-r--r-- 1 nobody nobody 14470610944 Jul  1 22:18 oldOutput.tar.zip

The files I'm backing up are all huge text files filled with numbers, like a huge comma delimited spreadsheet). Something like this

0.3454915028125262743685653,0.5590169943749474512628694,...
0.221761776923297210251107,0.3588180924674668759166707,...
-0.06101864995889930837202897,-0.09873024958113109372792593,...
-0.3001958820500086333460388,-0.4857271404396689140625654,...
...

Why is the tar file bigger than the directory? It should be compressed because I'm using the data with the z option. What's the point of tarring it, then?

share|improve this question
1  
You may already realize this, but the correct extension is .tar.gz. Read this. –  Evan Teitelman Jul 2 '13 at 2:32
4  
As a theoretical note: it's easy to show that, given any data-compressing algorithm, there exists texts such that the resulting output is at least as long as the original text. Otherwise, given a string of any length, X you could compress it to Y, a string of length strictly less than the length of X. This means that any file could be compressed, by multiple applications of the algorithm, to the empty file, which is absurd since data-compressing algorithms must be bijective/reversible. Hence it's not strange for archives to have bigger size than the original data. –  Bakuriu Jul 2 '13 at 7:46
add comment

4 Answers

up vote 12 down vote accepted

Your compressed tar file is smaller than its contents.

ls prints file sizes in bytes by default.
du -k prints file sizes in kilobytes.

0610944B ≈ 14131456KB < 77063332KB

To make ls print file sizes in kilobytes, use the -k flag.

share|improve this answer
    
I guess I could complain about Linux not using a "k" unit symbol or be happy that I have an answer - either way you are correct. The final tard file size, using ls -k oldOutput.tar.zip is 32703224 - less than half the original. –  Jeff Jul 2 '13 at 20:51
    
You can use the -h flag with both du and ls to get human-readable sizes. If you would like that to be your default, add a couple of aliases to your interactive shell config (.bashrc, .zshrc, etc): alias du='du -h' and alias ls='ls -h'. –  Evan Teitelman Jul 2 '13 at 21:24
add comment

Remember if you compress primary BINARY data (i.e. *.gz, *.zip) it is possible/probable that you will get an output file that is FAR LARGER than the original aggregation. So I'd lose the -z switch on the tar you are trying.

share|improve this answer
    
The OP is compressing text, not binary data. The text happens to be numbers but it is still text: "The files I'm backing up are all huge text files filled with numbers". –  terdon Jul 2 '13 at 3:27
    
See the command: tar -cvzf ../oldOutput.tar.zip *, it is compressing via tar -z an already zipped and compressed file which at that point IS binary data! –  mdpc Jul 2 '13 at 16:00
    
mdpc: Please clarify. What do you mean I am compressing already zipped files. It seems to me that @terdon's original comment is correct. –  Jeff Jul 2 '13 at 20:47
    
@Jeff you're quite right, I misread the command when I agreed with @mdpc. Your command is correctly creating oldOutput.tar.zip from (presumably) uncompressed files. –  terdon Jul 2 '13 at 20:55
    
tar -cvf created a much larger file than tar -cvzf - but it ran faster. –  Jeff Jul 6 '13 at 20:49
add comment

Text files don't compress better just because they labeled with a "txt" extension. Text files often compress better because there tends to be a lot of extra "white space" and duplicated letter usage.

I postulate that your CSV file has little to no "white space" to clean up and actually emulates a binary or graphics image file instead.

share|improve this answer
    
There is no white space at all in my CSV files, unless new lines are considered whitespace - probably not - in which case each file has exactly 10,000 white space characters. –  Jeff Jul 2 '13 at 20:49
add comment

Maybe you backup sparse files without the tar option --sparse?

You can easily find out by extracting the archive after it has been finished and comparing the source directory with the extracted directory.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.