Tag Info

New answers tagged

2

Here's another script. You can choose whether you want precisely one million files per segment, or precisely 30 segments. I've gone with the former in this script, but the split keyword allows either choice. #!/bin/bash # DIR="$1" # The source of the millions of files TARDEST="$2" # Where the tarballs should be placed # Create the million-file ...


3

I wrote this bash script to do it. It basically forms an array containing the names of the files to go into each tar, then starts tar in parallel on all of them. It might not be the most efficient way, but it will get the job done as you want. I can expect it to consume large amounts of memory though. You will need to adjust the options in the start of the ...


0

I might try extracting the data files after booting back to windows using windows tar. If tar omitted some files while you were creating the archive, it probably would have given some error messages. Also, you can list the contents of a tar file with: tar tvf tarfile.tar.gz


0

I cannot reproduce the results that you are getting, although I can get tar to throw an error when doing what you describe. Ignoring that, providing a list with spaces before a file name is incorrect; whitespaces are not insignificant in Unix. If you tell tar that a file is called "  file2" then tar looks for a file called "  file2" and ...


0

All of your problems are explained in Why does my shell script choke on whitespace or other special characters?, but I'll repeat the relevant parts here. First, $var in a shell script does not mean “the value of the variable var”, it means “take the value of the variable var, split it into words and interpret each word as a wildcard pattern”. To avoid these ...


3

This is not possible as of now with GNU tar, but things exist : https://github.com/stfp/tar-dedup : a patched GNU tar with an experimental --dedup-filter command http://zbackup.org/ which eats and spits tar formatted streams and dedupes Note that hardlinking and deduping does not have the same semantics, one would need another kind of tar node type to ...


0

You may be inadvertently shell-expanding the contents of your archive. Try quoting your variable. Replace echo $ARCHIVE_CONTENTS | tar xzv -C $TMPDIR with echo "$ARCHIVE_CONTENTS" | tar xzv -C $TMPDIR You also may need to avoid appending a newline with echo -n


1

why not use a here document like in the old days ? #!/bin/bash export TMPDIR=`mktemp -d /tmp/selfextract.XXXXXX` ## other stuff base64 -d <<EOF | tar xvf -C $TMPDIR ## base64 encoded EOF you produce base64 encoded this way tar cf - my_dir | base64 > /some/place (including it int self extract archive is left to the reader)


-1

This isn't a security risk. It's a safety feature at best. If you extract tarballs without prior checking the contents, you don't belong on a server anyway. Because then you are the security risk.


0

With zsh: tar cf foo.tar *(m-1)


1

No, it's not possible. See Wikipedia for a description of the format of a tar file. Basically, it's just an alternating sequence of file header and file contents. There's no table of contents at the beginning, all the information about each file is in its file header. The header includes the file size, so when just listing the contents the reader may be ...


2

The short answer is "no." 7Zip, RAR, zip, et al., are all dual-function programs/file formats. They compress individual files, then archive the compressed results. That results in portions of the file that are not compressed, but that, in turn, allows programs to get to individual files. gzip (the .gz part) is different. It compresses a single byte stream ...


0

Please find below the examples of extracting specific files from tar.gz file. From local file: $ tar xvf file.tgz path/README.txt 2nd_file.txt From remote URL: $ curl -s http://example.com/file.tgz | tar xvf - path/README.txt 2nd_file.txt


2

Use the -C option to change to the directory before archiving the file. tar cf file.tar -C /home/albertserres/Descargas file In Python this should be: saveFolder = "/home/albertserres/a.tar" srcFolder = "/home/albertserres/Descargas" srcFile = "test1" subprocess.call(["tar", "rvf", saveFolder, "-C", srcFolder, srcFile])


1

Ended up with HPN-SSHand pigz. tar -cf - -C [RELATIVEFOLDER] [FILENAME] | pigz | ssh px "pigz -d | tar xf - -C [REMOTEFOLDER]" Improvement by power of ten. For reference, installing HPN-SSH and pigz on Ubuntu 14.04 is easy as: # hpn-patched ssh from ppa sudo apt-get install python-software-properties sudo add-apt-repository ppa:w-rouesnel/openssh-hpn ...


0

Well there is zipsplit(1) and split(1), that you could used to split up the large files before using say scp(1) to copy them over. Or you could split the output into named pipes and then cat the named pipes to ssh, if you wish a pipelined effect.


-1

To add to the other answers, a single ? will translate to a single character filename and ?? will match filenames that has only two characters and so on. [root@mercy testdir_2]# ls ion it r [root@mercy testdir_2]# ls ? r [root@mercy testdir_2]# ls ?? it [root@mercy 1 testdir_2]# ls ??? ion [root@mercy testdir_2]#


1

The rm -R will actually delete all the files (if it has enough permissions). If you want to leave only the currently opened files, you can check with lsof +D /srv/log/prod/dms/: it will list all the files located under /srv/log/prod/dms/ and that are currently opened. Be careful, to script that, you should be sure that your daemon keeps its files handlers ...


1

wget -qO - |gzip -c > file_name.gz -c for stdout . that's '>' used. the file get from wget serialize to file_name.gz using standard output library. -qO to send output file


13

Do you really want to tar the file or are you looking for downloading a file into a compressed form. Tarring a file is just bundling (uncompressed) files into an (uncompressed) archive. If you want to download a file into a compressed file you can use: wget -qO - <url>|gzip -c - > file.gz


4

This is not going to work the way you want it to. A file (obviously with a filename) needs to be stored in the tar. That bit (the filename) is obviously missing if you just pipe the contents of the download to tar. I don't see any way to tell tar that it should pack stdin and specify a filename for that. That said, I really do not say a way to achieve that ...


8

You can't deal with input streams in that way. It is designed to deal with files. If you had managed to create an archive as you describe, what would it look like? How would you untar it? There would be no filename to create, just data. I think your best bet is to get the file and tar it in two separate commands. If you don't want the file to remain on ...



Top 50 recent answers are included