191

I have to download a file from this link. The file download is a zip file which I will have to unzip in the current folder.

Normally, I would download it first, then run the unzip command.

wget http://www.vim.org/scripts/download_script.php?src_id=11834 -O temp.zip
unzip temp.zip

But in this way, I need to execute two commands, wait for the completion of first one to execute the next one, also, I must know the name of the file temp.zip to give it to unzip.

Is it possible to redirect output of wget to unzip? Something like

unzip < `wget http://www.vim.org/scripts/download_script.php?src_id=11834`

But it didn't work.

bash: `wget http://www.vim.org/scripts/download_script.php?src_id=11834 -O temp.zip`: ambiguous redirect

Also, wget got executed twice, and downloaded the file twice.

Improve this question
2
  • In the latter example, wget probably was executed twice because the ? is a special character in the shell. Putting the URL in ""s should help.
    – p-static
    Commented Oct 8, 2010 at 0:32
  • This thread seems to have a solution. Haven't tried it myself though. serverfault.com/questions/26474/…
    – user37850
    Commented Apr 25, 2013 at 13:00

9 Answers 9

Reset to default
138

You have to download your files to a temp file, because (quoting the unzip man page):

Archives read from standard input are not yet supported, except with funzip (and then only the first member of the archive can be extracted).

Just bring the commands together:

wget "http://www.vim.org/scripts/download_script.php?src_id=11834" -O temp.zip
unzip temp.zip
rm temp.zip

But in order to make it more flexible you should probably put it into a script so you save some typing and in order to make sure you don't accidentally overwrite something you could use the mktemp command to create a safe filename for your temp file:

#!/bin/bash
TMPFILE=`mktemp`
PWD=`pwd`
wget "$1" -O $TMPFILE
unzip -d $PWD $TMPFILE
rm $TMPFILE
Improve this answer
3
  • 1
    Is wget file.zip && unzip file.zip the same as wget file.zip; unzip file.zip or is one preferred over the other? Thanks :)
    – jaggedsoft
    Commented Dec 3, 2016 at 20:10
  • 12
    @NextLocal wget && unzip will run unzip only if wget succeeded. wget ; unzip will run unzip anyway, possibly pointing to non existent file.
    – temoto
    Commented Feb 7, 2017 at 11:37
  • 3
    funzip was the answer I was looking for. Terraform (for some reason) packages it's binary as a single file in a zip archive so this was perfect for me.
    – Asfand Qazi
    Commented Nov 7, 2018 at 22:52
121

This is a repost of my answer to a similar question:

The ZIP file format includes a directory (index) at the end of the archive. This directory says where, within the archive each file is located and thus allows for quick, random access, without reading the entire archive.

This would appear to pose a problem when attempting to read a ZIP archive through a pipe, in that the index is not accessed until the very end and so individual members cannot be correctly extracted until after the file has been entirely read and is no longer available. As such it appears unsurprising that most ZIP decompressors simply fail when the archive is supplied through a pipe.

The directory at the end of the archive is not the only location where file meta information is stored in the archive. In addition, individual entries also include this information in a local file header, for redundancy purposes.

Although not every ZIP decompressor will use local file headers when the index is unavailable, the tar and cpio front ends to libarchive (a.k.a. bsdtar and bsdcpio) can and will do so when reading through a pipe, meaning that the following is possible:

wget -qO- http://example.org/file.zip | bsdtar -xvf-
Improve this answer
4
  • 2
    This is excellent! I would note that tar gives me some warnings about the uncompressed data being the wrong size (expected 0), but the files themselves appear to be undamaged. Guessing this is due to the lack of the index.
    – Wyatt Ward
    Commented Mar 17, 2018 at 7:50
  • 3
    I have a .zip-file here that contains files with executable permissions. When I download and pipe into bsdtar, the exec bits get thrown away. When I download to disk and extract with bsdtar or unzip then, the exec bits are honoured.
    – Golar Ramblar
    Commented May 4, 2018 at 16:02
  • // , @GolarRamblar, didst ever find out why?
    – Nathan Basanese
    Commented Sep 27, 2018 at 19:16
  • 2
    @NathanBasanese: here is the answer. In short: A ZIP archive has two places where it stores such information, that can be inconsistent, and depending if the file bsdtar opens is seekable or not it uses the one or the other place.
    – Golar Ramblar
    Commented Dec 11, 2018 at 15:54
34

If you have the JDK installed, you can use jar:

wget -qO- http://example.org/file.zip | jar xvf /dev/stdin
Improve this answer
6
  • 9
    I just found that jar doesn't preserve file permissions. Nice trick otherwise.
    – phunehehe
    Commented Dec 25, 2016 at 10:15
  • 13
    You don't need to give a file param, just use | jar xv
    – OneCricketeer
    Commented Aug 25, 2017 at 15:55
  • I, too, was bitten by the assumption than jar could be used as a replacement for unzip; Unfortunately jar does not restore extracted files' permissions;
    – cueedee
    Commented Mar 9, 2020 at 13:22
  • 2
    Just use | jar x
    – korakot
    Commented May 17, 2020 at 1:50
  • 1
    Does the pkzip format even store file permissions in the first place?
    – Johan Boulé
    Commented Dec 18, 2020 at 19:26
26

Repost of my answer:

BusyBox's unzip can take stdin and extract all the files.

wget -qO- http://downloads.wordpress.org/plugin/akismet.2.5.3.zip | busybox unzip -

The dash after unzip is to use stdin as input.

You can even,

cat file.zip | busybox unzip -

But that's just redundant of unzip file.zip.

If your distro uses BusyBox by default (e.g. Alpine), just run unzip -.

Improve this answer
1
  • 1
    further to @Saftever's answer, which i'm not allowed to comment on, busybox will work but versions older than 1.27.0 won't due to a redundant lseek, see changelog busybox.net
    – Andrzej Cichocki
    Commented Dec 13, 2020 at 16:43
20

I don't think you even want to bother piping wget's output into unzip.

From the wikipedia "ZIP (file format)" article:

A ZIP file is identified by the presence of a central directory located at the end of the file.

wget has to completely finish the download before unzip can do any work, so they run sequentially, not interwoven as one might think.

Improve this answer
18

The proper syntax would be:

$ unzip <(curl -sL https://www.winpcap.org/archive/1.0-docs.zip)

but it won't work, because of the error (Info-ZIP on Debian):

lseek(3, 0, SEEK_SET)                   = -1 ESPIPE (Illegal seek)

Archive:  /dev/fd/63
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /dev/fd/63 or
        /dev/fd/63.zip, and cannot find /dev/fd/63.ZIP, period.

or on BSD/OS X:

Trying to read large file (> 2 GiB) without large file support

This is, because the standard zip tools are mainly using lseek function in order to set the file offset at the end to read its end of central directory record. It is located at the end of the archive structure and it is required to read the list of the files (see: Zip file format structure). Therefore the file cannot be FIFO, pipe, terminal device or any other dynamic, because the input object cannot be positioned by the lseek function.

So you have the following workarounds:

  • use different kind of compression (e.g. tar.gz),
  • you have to use two separate commands,
  • use alternative tools (as suggested in other answers),
  • create an alias or function to use multiple commands.
Improve this answer
1
  • I think it still could be a FIFO. You'd just have to keep reading from the FIFO until EOF (effectively buffering the whole FIFO in memory or in a temp file). Totally doable to ease script creation, but not very useful.
    – Evan Carroll
    Commented Oct 9, 2016 at 20:28
3

If there is only one file in zip, you can use zcat or gunzip or funzip :

wget -qO- http://www.vim.org/scripts/download_script.php?src_id=11834 | funzip -

To unzip more that one file inside a zip archive provided from a pipe, you can either wait for the unzip team to implement it or use the busybox unzip solution from Saftever.

Just for information : Here are the definitions of gunzip and zcat on my system :

$ grep ^exec $(which gunzip zcat)
/bin/gunzip:exec gzip -d "$@"
/bin/zcat:exec gzip -cd "$@"
Improve this answer
2

This works for me quite well on macOS :

tar xvf <(curl -sL http://www.vim.org/scripts/download_script.php?src_id=11834)

jar xvf <(curl -sL http://www.vim.org/scripts/download_script.php?src_id=11834)

wget -qO- http://www.vim.org/scripts/download_script.php?src_id=11834 | tar xvf -

wget -qO- http://www.vim.org/scripts/download_script.php?src_id=11834 | jar xvf -

BTW : The tar version used here is :

$ tar --version
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6
Improve this answer
6
  • The tar tool on my system says: tar: This does not look like a tar archive. I use tar (GNU tar) 1.30 on ubuntu/debian. Can you please add the tar --version output of the tar tool you have used your system to type these commands in your answer ?
    – SebMa
    Commented Dec 5, 2021 at 12:03
  • 1
    @SebMa I'm using latest macOS and all these commands works for me. My tar --version command output is: bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6
    – Maksim Kostromin
    Commented Dec 8, 2021 at 13:23
  • @SebMa Output for both (1st and 3rd line of code) is same for me: x doc/NERD_tree.txt x nerdtree_plugin/exec_menuitem.vim x nerdtree_plugin/fs_menu.vim x plugin/NERD_tree.vim
    – Maksim Kostromin
    Commented Dec 8, 2021 at 13:23
  • You are using bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6 on macOS. On Linux, we have GNU tar which has a bit different behavior.
    – SebMa
    Commented Dec 8, 2021 at 18:53
  • well, but anyway, it's not a reason to delete my answer) It worked on latest macOS environment what I'm also mentioned
    – Maksim Kostromin
    Commented Dec 9, 2021 at 14:13
0

A zip archive is not sequential because it often has the table of contents at the end of the file, so it is difficult to stream-unzip it.

An alternative solution is to see if you can get another file format, like .tar.gz.

For example, if you're downloading a .zip file from GitHub, there is almost always a .tar.gz version available.

For example,

Notice the pattern -- just replace .zip with .tar.gz and pipe to | tar xzf -

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .