I have a large file
foo.tar.xz that contains a lot (say 200000) of files. I figured out that this archive contains some (around 5000) files I don't want. I don't have sufficient disk space to decompress the whole thing onto my disk; additionally, I fear attributes / rights might get lost if I do so. I have enough space to host two copies of the compressed archive though. Is there a tool to remove some of the files from the archive (specified with a regex on the filename) on-the-fly, i.e. without unpacking the archive into individual files?
I have a large file
GNU tar has a
--delete option that works with archives too nowadays.
Use it like this, for example:
tar -vf yourArchive.tar --delete your/path/to/delete
Beware: It will most likely not work on any kind of magnetic tape medium. But
tar has no problems working in a pipe, so you can just use a temporary tar file and overwrite the tape with that afterwards. It also won't work on compressed files, so you would need to uncompress the file.
Also, the operation will be rather slow in any case, due to the (by design) packed linear nature of tar archives.
1It does exist, but it doesn't work with files where random access is not possible (e.g. compress archives) but this is my use-case. Aug 31, 2016 at 12:18
1The other problem is that I cannot specify a pattern to delete. Note my comment from 2013 where I already address the shortcomings of
gtar --delete. Aug 31, 2016 at 12:19
--wildcardsallows you to use patterns rather than filenames, so create a temporary file containing the patterns and use
unxz < file.tar.xz | tar --wildcards --delete -T patternfile | xz > file2.tar.xz. It won't do a full regex (if you need that, just use
tar -tand build up a list of filenames to delete), just filename matching patterns. Aug 31, 2016 at 14:00
1BEWARE: this command might corrupt your tar file. Unfortunately, it destroyed mine and I was dumb enough to not create a backup copy. I'm not sure what the reason is, but in my case, it started creating thousands of duplicates for every file. I had to SIGTERM the process because the archive grew 10x from the original size, but the data has been already lost by that moment.– noomorphApr 10, 2022 at 10:49
(edited, as I misunderstood the question, which was since edited also)
The best you can do is to extract, delete, and recompress the entire file.
unxz < foobar-old.tar.xz | tar --delete foo/bar | xz > foobar-new.tar.xz
It's not possible to delete files from a tar directly.
tar is a stream, originally intended for tape drives which do not do random seeks well - while in theory it could be possible on a disk filesystem to punch a hole / rewrite the remaining file, with compression the point is moot as most if not all compression methods heavily depend on contents that occured earlier in the file. In order to do this in place you would need very detailed knowledge about both the compression method as well as the tar file format. That's complexity to a point no one would even bother with it. It's cheaper to just keep the files around and ignore them.
If you need this functionality, tar is probably not what you want.
Those files make up 35% of the archives size. The restrictions you point out seemingly do only apply if I rewrite the file, not if I modify it out-of-place, which I can do (I have enough space to save the packed archive twice). Is there such a tool? Mar 21, 2013 at 20:57
I may have misunderstood your question then. If you ARE willing to unpack the tar after all, and repack it, (just without actually creating the tarred files - i.e., a direct tar to tar pipe), it may be possible. Mar 21, 2013 at 21:01
Yeah, I can do that. It's just that the files have uids/gids/attributes that I need to preserve. Also, I do not have enough disk space to save the unpacked representation. I have enough space to save two packed archives though. Mar 21, 2013 at 21:05
1That's no problem at all. If I can do this in one pass, the time won't be too long. I can't imagine any archive format that allows for fast deletation while actually releasing storage. Mar 21, 2013 at 21:20
--wildcardshelp... I had to include
./at the start of the pattern though... Mar 5, 2018 at 12:59
According to the manual, you can pass a list of filenames to
tar to only extract those. For example:
$ tar --file archive.tar --list foo bar baz $ tar --file archive.tar --extract foo
I don't see how --extract helps me. Could you elaborate? Please keep in mind that I cannot unpack the archive (or substantial parts of it) to disk. Mar 21, 2013 at 21:07
2Please do not just post links: this is a wiki--add sufficient content for it to be unnecessary for people to leave the page to understand your answer. Mar 21, 2013 at 22:02