Hot answers tagged

22

The first step is to figure out what the bottleneck is: is it disk I/O, network I/O, or CPU? If the bottleneck is the disk I/O, there isn't much you can do. Make sure that the disks don't serve many parallel requests as that can only decrease performance. If the bottleneck is the network I/O, run the compression process on the machine where the files are ...


12

The ._ files are how OS X bsdtar handles OS X-specific extended attributes and resource forks. To keep them from being added to your tar files, you can pass COPYFILE_DISABLE=1 as an environment variable to tar. COPYFILE_DISABLE=1 tar cf newTar.tar /your/files


12

You can install pigz, parallel gzip, and use tar with the multi-threaded compression. Like: tar -I pigz -cf file.tar.gz * Where the -I option is: -I, --use-compress-program PROG filter through PROG Of course, if your NAS doesn't have multiple cores / powerful CPU, you are limited anyway by the CPU power. The speed of the hard-disk/array on which the ...


6

Archive::Tar or similar software would be one method. % touch foo % tar cvf x foo foo % tar tvf x -rw-rw-r-- 1 jdoe12 jdoe12 0 May 6 20:36 foo % perl -MArchive::Tar -e '$t=Archive::Tar->new;$t->read("x");$t->chown("foo","root");$t->write("y")' % tar tvf y -rw-rw-r-- 1 root jdoe12 0 May 6 20:36 foo %


5

There are several issues with your command. The first one is a compressed archived cannot be updated. You would need to first create a uncompressed archive, update it and finally compress it. The second one is the tar syntax is incorrect, the first -exec clause should use either the + terminator or the \; one, but not both. The third one is more subtle ...


5

To my understanding, tar --exclude='._*' -cvf newTar . should work: Finder creates the ._* files but newTar shouldn't contain them. But you can completely bypass those files by invoking tar in passthrough mode. For example, to copy only the files from oldTar that are under some/path, use tar -cf newTar --include='some/path/*' @oldTar


4

You asked tar to archive the files file2 and total.tar in the archive called file1, which it attempted to do. Unfortunately that means that file1 was overwritten, all you can get from it now is file2: tar tvf file1 (don't add a z in there, you didn't specify it when creating the archive). The only way you'll recover file1 is from backups.


4

tar itself does not have this feature, you could only filter it so you have two independent tar files, one holding the compressed files, another for the uncompressed files. This can be achieved using tar's exclude option you mentioned, or by building file lists using find. If you are open to alternatives, there is dar (disk archive) which can decide whether ...


3

No, you can't update a compressed archive using tar. But, if your script is creating the archive (using touch), you can update it and compress it later. Change rjvf to rvf, and at the end of your script, run bzip2 archive.tar (I take it from the j option and your tags that you want to use bzip2 for compression).


3

You cannot change permissions explicitly on extraction, but you can put the preferred identities into the tarball on creation (even when not running as root). tar cvf /tmp/tarball.tar --user=naftuli --group=othergroup files... If the source system doesn't know your account details you can suffix the names with the preferred uid and gid (e.g. ...


3

tar can cope with partial archives after splitting. When you try to restore part of such an archive, it will skip over whatever it can't use at the start, and tell you about any partial file at the end; everything in between will be restored properly. You can instruct tar itself to split archives as it creates them, using the tape length options; see Create ...


2

As others have identified, the problem with your command is that it includes directories, and tar archives them recursively. If a directory has been modified recently, all the files in it and its subdirectories get included, whether they have been modified or not. If you don't care to back up directory metadata, then just tell find not to print directory ...


2

You will not only run into problems if xargs invokes tar twice, you also will get problems if your file-names contain special characters like newlines. You should drop the use of xargs and tar and use find with cpio: find $SOURCEDIR -mtime -1 -print0 | cpio --create -0 --verbose \ --format=ustar -O $ARCHIVE ustar provides you with a POSIX.1 ...


2

find $SOURCEDIR -mtime -1 also includes $SOURCEDIR in the output, which needs to be removed before further processing Using grep -vx one can define a particular line to be excluded.. find $SOURCEDIR -mtime -1 -print | grep -xv "$SOURCEDIR" | xargs -r tar cvf $ARCHIVE || { echo "No files have been changed in the past 24 hours. Exiting script ..." ; exit ...


2

Those files starting with "._*" are apple specific location indicator files according to THIS POST and you obviously can not get rid of them while logged in to your terminal om OSX, again according to the same page. You need to upload the file to a non-apple OS, get rid of those files and tar them up again. This seems to be the only solution.


2

You can reduce the amount of compression (in terms of space saved) to make it faster. To start with, bzip2 is MUCH slower than gzip, though it compresses smaller. You can also change the compression level of bzip2, gzip, or most compression programs to trade size for speed. If you are not willing to trade size of speed, you can still probably get the same ...


2

You are using printf to escape the spaces, as well as quoting it "" If you omit the printf call, and use the original $d with double-quotes, you are good. f=`echo $d | tr ' ' '_' | tr -d ',.!'` # Clean the name t=$YEAR_DIR/$f.tar.gz ##d=$(printf '%q' "$d") escapes the spaces # echo tar czf $t "$d" # This outputs the command as expected ...


2

No, in general this isn't possible. A FTP server usually has commands to get information about files and directories and to store, retrieve, delete and rename files. Commands to mount devices and to send messages to users are also standardized but not implemented in current servers. See the list of FTP commands on Wikipedia for details. No RFC mentions a ...


2

By far the fastest and most effective way of compressing data is to generate less of it. What kinds of logs are you generating? 200GB daily sounds like quite a lot (unless you're google or some ISP...), consider that 1MB of text is about 500 pages, so you're generating the equivalent of 100 million pages of text per day, you'll fill the library of congress ...


2

If the only requirement is that the compression is fast, I'd recommend lz4 very highly. It's used in lots of places where the speed of compression is more important than the compression ratio (e.g. filesystems with transparent compression like ZFS)


2

No, you can't. At least not directly. tar doesn't do any compression. It merely reads part of the (virtual) file system, and generates one cohesive stream from it. This stream is then often passed to a compression tool/library, for instance gzip/libz. The compression part does not see or even know about individual files. It just compresses the stream ...


2

I don't know any way of doing this even with GNU tar. tar collects the files and then compresses the resulting stream (it's actually the equivalent of tar ... | gzip > somefile.tgz so by the time the compression is applied there is no awareness of individual files.


1

Any existing tarballs can be sanitized via something like: #!/usr/bin/env perl use strict; use warnings; use Archive::Tar 1.80; for my $tarfile (@ARGV) { my $tar = Archive::Tar->new($tarfile) // die "failed to read '$tarfile'\n"; for my $archivefile ($tar->list_files) { $tar->chown($archivefile, 'root:root'); } my ...


1

Generally .tar.gz is a usable file distribution format. GNU tar allows you to replace owner, group and permissions with other values when adding files to the archive. $ tar -c -f archive.tar --owner=0 --group=0 . https://www.gnu.org/software/tar/manual/html_section/tar_33.html#SEC69 If your version of tar does not support the GNU options you can copy ...


1

With GNU you can use --numeric-owner to prevent tar from storing your username. Alternatively, you can set another username with --owner=NAME. But you'll always get a user id in there. When it's extracted, those user ids will be dropped, unless the extractor is the root user. A common way used to bundle files is cpio which is typically used with the ...


1

Unless it is a vendor provided package that integrates with the OS package manager you shouldn't extract it in that location. Please go to /opt and extract it there for example and set up your environments variables as shown on the download page for Swift. By doing a full extraction in /opt, which is there for third-party applications, you can also quickly ...



Only top voted, non community-wiki answers of a minimum length are eligible