Say I have a file of 80GB /root/bigfile on a 100GB system and want to put this file in a archive /root/bigarchive.tar

I obviously need to delete this file at the same time that it is added in the archive. Hence my question:

How to delete a file at the same time that it is added in an archive?


If you're using GNU tar command, you can use the --remove-files option:


remove files after adding them to the archive

tar -cvf files.tar --remove-files my_directory
  • 5
    I think the OP wants to remove the file at the same time it is archived, so if --remove-files removes after adding the file to the .tar, it won't be helpful for him since his hard disk would be out of space. – Zumo de Vidrio Jan 4 '17 at 13:27

An uncompressed tar archive of a single file consists of a header, the file, and a trailing pad. So your principle problem is how to add 512 bytes of header to the start of your file. You can start by creating the wanted result with just the header:

tar cf - bigfile | dd count=1 >bigarchive.tar

Then copy the first 10G of your file. For simplicitly we assume your dd can read/write 1Gib at a time:

dd count=10 bs=1G if=bigfile >>bigarchive.tar

We now deallocate the copied data from the original file:

fallocate --punch-hole -o 0 -l 10GiB bigfile

This replaces the data with sparse zeroes that take no space on the filesystem. Continue in this manner, adding a skip=10 to the next dd, and then incrementing the fallocate starting offset to -o 10GiB. At the very end add some nul characters to pad out the final tar file.

If your filesystem does not support fallocate you can do something similar, but starting at the end of the file. First copy the last 10Gibytes of the file to an intermediate file called, say, part8. Then use the truncate command to reduce the size of the original file. Proceed similarly until you have 8 files each of 10Gibyte. You can then concatenate the header and part1 to bigarchive.tar, then remove part1, and then concatenate part2 and remove it, and so on.


Deleting a file does not necessarily do what you think it does. That's why in UNIX-like systems the system call is called unlink and not delete. From the manual page:

unlink() deletes a name from the filesystem.  If that name was the last
link to a file and no processes have the file open, the file is deleted
and the space it was using is made available for reuse.

If the name was the last link to a file but any processes still have
the file open, the file will remain in existence until  the  last  file
descriptor referring to it is closed.

As a consequence, as long as the data compressor / archiver is reading from the file, that file remains in existence, occupying space in the filesystem.


How to delete a file at the same time that it is added in an archive?

Given the context, I'll interpret this question as:

How to remove data from the disk immediately after it was read, before the full file has been read, so that there is enough space for the transformed file.

The transformation can be anything you want to do with the data: compressing, encrypting, etc.

The answer is this:

<$file gzip | dd bs=$buffer iflag=fullblock of=$file conv=notrunc

In short: read data, throw it into gzip (or whatever you want to do with it), buffer the output so we are sure to read more than we write, and write it back to the file. This is a version that's prettier and shows output while running:

cat "$file" \
| pv -cN 'bytes read from file' \
| gzip \
| pv -cN 'bytes received from compressor' \
| dd bs=$buffer iflag=fullblock 2>/dev/null \
| pv -cN 'bytes written back to file' \
| dd of="$file" conv=notrunc 2>/dev/null

I'll go through it, line by line:

cat "$file" reads the file you want to compress. It's a useless use of cat (UUOC) since the next part, pv, can also read the file, but I find this to be prettier.

It pipes it into pv which shows progress information (-cN tells it 'use some sort of [c]ursor' and give it a [N]ame).

That pipes into gzip which obviously does the compression (reading from stdin, outputting to stdout).

That pipes into another pv (pipe view).

That pipes into dd bs=$buffer iflag=fullblock. The $buffer variable is a number, something like 50 megabytes. It's however much RAM you want to dedicate to the safe handling of your file (as a data point, 50MB buffer for a 2GB file was fine). The iflag=fullblock tells dd to read up to $buffer bytes before piping it through. In the beginning, gzip will write a header, so gzip's output will land in this dd line. Then dd will wait until it has enough data before piping it through, and so the input can read further. Furthermore, if you have uncompressible parts, the output file might be bigger than the input file. This buffer makes sure that, up to $buffer bytes, this is not an issue.

Then we go into another pipe view line, and finally onto our output dd line. This line has of (output file) and conv=notrunc specified, where notrunc tells dd not to truncate (delete) the output file before writing. So if you have 500 bytes of A and you write 3 bytes of B, the file will be BBBAAAAA... (instead of be replaced by BBB).

I did not cover the 2>/dev/null parts, and they're unnecessary. They just tidy up the output a little by suppressing dd's "I'm finished and wrote this many bytes" message. The backslashes at the end of each line (\) make bash treat the whole thing as one big command that pipes into each other.

Here is a full script for easier use. Anecdotally, I put it in a folder called 'gz-in-place'. I then realized the acronym I made: GZIP: gnu zip in-place. So hereby I present, GZIP.sh:

#!/usr/bin/env bash

### Settings

# Buffer is how many bytes to buffer before writing back to the original file.
# It is meant to prevent the gzip header from overwriting data, and in case
# there are parts that are uncompressible where the compressor might exceed
# the original filesize. In these cases, the buffer will help prevent damage.
buffer=$((1024*1024*50)) # 50 MiB

# You will need something that can work in stream mode from stdin to stdout.

# For gzip, you might want to pass -9 for better compression. The default is
# (typically?) 6.

### End of settings

# FYI I'm aware of the UUOC but it's prettier this way

if [ $# -ne 1 ] || [ "x$1" == "x-h" ] || [ "x$1" == "x--help" ]; then
    cat << EOF
Usage: $0 filename
Where 'filename' is the file to compress in-place.

Only operate on data that you have backups of.
(But you always back up important data anyway, right?)

See the source for more settings, such as buffer size (more is safer) and
compression level.

The only non-standard dependency is pv, though you could take it out
with no adverse effects, other than having no info about progress.
    exit 1;

echo "Progressing '$1' with ${b}MiB buffer...";
echo "Note: I have no means of detecting this, but if you see the 'bytes read from";
echo "file' exceed 'bytes written back to file', your file is now garbage.";
echo "";

cat "$1" \
| pv -cN 'bytes read from file' \
| $compressor $compressorargs \
| pv -cN 'bytes received from compressor' \
| dd bs=$buffer iflag=fullblock 2>/dev/null \
| pv -cN 'bytes written back to file' \
| dd of="$1" conv=notrunc 2>/dev/null

echo "Done!";

I feel like adding another buffering line before gzip, to prevent it from writing too far when the buffering dd line flushes through, but with only 50MiB buffer and 1900MB of /dev/urandom data, it seems to work already anyway (the md5sums matched after decompressing). Good enough ratio for me.

Another improvement would be detection of writing too far, but I don't see how to do that without removing the beauty of the thing and creating a lot of complexity. At that point, you might as well just make it a fully-fledged python program that does it all properly (with failsafes to prevent data destruction).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.