Tag Info

Hot answers tagged

30

Compression ratio is very dependent of what you're compressing. The reason text compresses down so well is because it doesn't even begin to fully utilize the full range of numbers representable in the same binary space. So formats that do (e.g compressed files) can store the same information in less space just by virtue of using all those binary numbers that ...


29

Globs are not regular expressions. In general, the shell will try to interpret anything you type on the command line that you don't quote as a glob. Shells are not required to support regular expressions at all (although in reality many of the fancier more modern ones do, e.g. the =~ regex match operator in the bash [[ construct). The .??* is a glob. ...


27

XZ_OPT=-9 tar cJf tarfile.tar.xz directory tar's lowercase j switch uses bzip, uppercase J switch uses xz. The XZ_OPT environment variable lets you set xz options that cannot be passed via calling applications such as tar. This is now maximal See man xz for other options you can set.


26

There are several different patterns for options that have been used historically in UNIX applications. Several old ones, like tar, use a positional scheme: command options arguments as for example tar uses tar *something*f "file operated on" *"paths of files to manipulate"* In a first attempt to avoid the confusion, tar and a few other programs ...


20

I think you are looking for pbzip2: PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. Have a look at the project homepage or check your favorite package repository.


19

That's actually a feature, not a problem. Archives with absolute locations are a security risk. Attackers could use such archives to trick users into installing files in critical system locations. Yes, you could use -P. But what's wrong with allowing tar to remove the forward slash, and simply requiring the user of the archive to explicitly do the ...


18

It's a common convention to use - in a filename argument to mean stdin or stdout; tar follows the same convention. From the man page: -f, --file [HOSTNAME:]F use archive file or device F (default "-", meaning stdin/stdout)


18

tar is one of those ancient commands from the days when option syntax hadn't been standardized. Because all useful invocations of tar require specifying an operation before providing any file name, most tar implementations interpret their first argument as an option even if it doesn't begin with a -. Most current implementations accept a -; the only ...


17

This should work: mkdir pretty_name && tar xf ugly_name.tar -C pretty_name --strip-components 1 -C changes to the specified directory before unpacking (or packing). --strip-components removes the specified number of directories from the filenames stored in the archive. Note that this is not really portable. GNU tar and at least some of the BSD ...


15

You posted in a comment that you are working on a Mac OS X system. This is an important clue to the purpose of these ._* files. These ._* archive entries are chunks of AppleDouble data that contain the extra information associated with the corresponding file (the one without the ._ prefix). They are generated by the Mac OS X–specific copyfile(3) family of ...


15

tar -c data_dir | wc -c without compression or tar -cz data_dir | wc -c with gzip compression or tar -cj data_dir | wc -c with bzip2 compression will print the size of the archive that would be created in bytes, without writing to disk. You can then compare that to the amount of free space on your target device. You can check the size of the data ...


14

From man tar: -C directory In c and r mode, this changes the directory before adding the following files. In x mode, change directories after opening the archive but before extracting entries from the archive. i.e, tar xC /foo/bar -f /tmp/foo.tar.gz should do the job. (on FreeBSD, but GNU tar is basically the same in this ...


13

Simple example, creating a gzipped tarball out of a directory, excluding a subdirectory and listing the contents: $ ls fruit/ apple banana peach tomato $ tar czf onlyfruit.tar.gz --exclude=tomato fruit/ $ tar tf onlyfruit.tar.gz fruit/ fruit/peach/ fruit/apple/ fruit/banana/


12

7zip can run on multiple threads when given the -mmt flag, but only when compressing into 7z-archives which offer great compression but are generally slower to create then zip archives. Do something like this. 7z a -mmt foo.7z /opt/myhugefile.dat


12

I've just had exactly this problem. As Gilles suggested, upgrading tar is the answer but (surprise surprise) tar can't be upgraded in the usual way because dpkg requires version 1.23 or later before it'll unpack and install the latest tar deb. dpkg really needs an explicit dependency to ensure that when a later version of dpkg is installed, the latest tar ...


12

When you send the same set of files, rsync is better suited because it will only send differences. tar will always send everything and this is a waste of resources when a lot of the data are already there. The tar + rsync + untar loses this advantage in this case, as well as the advantage of keeping the folders in-sync with rsync --delete. If you copy the ...


12

If you want to exclude an entire directory, your pattern should match that directory, not files within it. Use --exclude=/data/sub1 instead of --exclude='/data/sub1/*' Be careful with quoting the patterns to protect them from shell expansion. See this example, with trouble in the final invocation: $ for i in 0 1 2; do mkdir -p /tmp/data/sub$i; echo foo ...


12

tar stores relative paths. GNU tar even says so if you try to store an absolute path: tar -cf foo.tar /home/foo tar: Removing leading `/' from member names If you need to extract a particular folder, have a look at what's in the tar file: tar -tvf foo.tar And note the exact filename. In the case of my foo.tar file, I could extract /home/foo/bar by ...


12

You compression attempts failed because your data is already highly compressed and there's not much more to gain, see the other answers for more detailed explanations. However, if you can agree on lossy compression, in contrast to lossless like you tried before, you can compress the images significantly. But since data is cut away, it can not be undone. ...


12

Your compressed tar file is smaller than its contents. ls prints file sizes in bytes by default. du -k prints file sizes in kilobytes. To make ls print file sizes in kilobytes, use the -k flag.


11

With GNU tar ≥1.16, use --transform to apply a sed regexp transformation to each file name (the transformation is applied on the full path in the archive): tar xf foo.tar --transform 's!^ugly_name\($\|/\)!pretty_name\1!' If ugly_name is the toplevel component of all the file names in the archive, you don't need to bother with precise matching: tar xf ...


10

Actually, I believe the question was not about the (standard) file permission bits, but extended ACL information (see setfacl(1) or acl(5)). To my knowledge, the unmodified GNU tar ignores ACL information. (The man page for GNU tar 1.15.1 as shipped with RHEL 5.2 mentions switches --acls and --no-acls, but I haven't gotten them to work.) However, the star ...


10

By name You can generate the list of files in the archive and delete them, though this is annoyingly fiddly with archivers such as unzip or 7z that don't have an option to generate a plain list of file names. Even with tar, this assumes there are no newlines in file names. tar tf foo.tar | while read -r file; do rm -- "$file" done unzip -l foo.zip | awk ' ...


10

This will list all the PDFs: $ find dir/ -name '*.pdf' ./dir/subdir2/subsubdir1/document.pdf ./dir/subdir3/another-document.pdf You can pipe that to xargs to get it as a single space-delimited line, and feed that to tar to create the archive: $ find dir/ -name '*.pdf' | xargs tar czf dir.tar.gz (This way omits the empty directories)


10

Tar with bzip2 compression should take as much load off the network and on the cpu. $ tar -C /path/to/src/dir -jcf - ./ | ssh user@server 'tar -C /path/to/dest/dir -jxf -' Not using -v because screen output might slow down the process. But if you want a verbose output use it on the local side of tar (-jcvf), not on the remote part. If you repeatedly copy ...


10

tar has an option to suppress this message [1]: -m, --touch don't extract file modified time However, you should probably also check that you don't have an issue with your system clock. [1] http://unixhelp.ed.ac.uk/CGI/man-cgi?tar


10

Have a look first at git help archive. archive is a git command that allows to make archives containing only git tracked files. Probably what you are looking for. One example listed at the end of the man page: git archive --format=tar --prefix=git-1.4.0/ v1.4.0 | gzip >git-1.4.0.tar.gz


10

You should check if most of your time are being spent on CPU or in I/O. Either way, there are ways to improve it: A: don't compress You didn't mention "compression" in your list of requirements so try dropping the "z" from your arguments list: tar cf. This might be speed up things a bit. There are other techniques to speed-up the process, like using "-N " ...


10

I may be a dinosaur, but I think that habitually using "cvf" instead of "-cvf" is probably more portable. I imagine most Linux distros use GNU tar, and I would guess that the *BSDs do also, but you'll find proprietary Unixes that still use the old SysV tar, which used to require you to not use a '-' in the options. I do not use "-cvf" (or "-xf" or ...


10

Most common image formats are already compressed (like jpg, png, gif), so you don't get much savings. 1% sounds about right. Adding more compression can actually make the result (slightly) larger, because the compression algorithm has no benefit on compressed data, and then the format (eg. gzip) has to add header and/or structure information to the output. ...



Only top voted, non community-wiki answers of a minimum length are eligible