4

I'm trying to decompress ~8GB .zip file piped from curl command. Everything I have tried is being interrupted at <1GB and returns a message:

... has more than one entry--rest ignored

I've tried: funzip, gunzip, gzip -d, zcat, ... also with different arguments - all end up in the above message.

The datafile is public, so it's easy to repro the issue:

curl -L https://archive.org/download/nycTaxiTripData2013/faredata2013.zip | funzip > datafile
5
  • Out of curiosity, have you encountered the issue if you try this as separate steps? (ie curl to save to file, then funzip)
    – haylem
    Jun 22, 2015 at 8:01
  • @haylem yes, when i just try to download it, curl stop at ~3.2GB...
    – Oleg Shirokikh
    Jun 22, 2015 at 8:45
  • @OlegShirokikh are you using 32 bit operating system?
    – taliezin
    Jun 22, 2015 at 8:52
  • @haylem - 64bit... I have got couple single-part .tar.gz files that are way larger than 8GB uncompressed
    – Oleg Shirokikh
    Jun 22, 2015 at 8:56
  • To get all the file, you can use curl -C - ... to restart a partial download to a file (then use unzip).
    – meuh
    Jun 22, 2015 at 10:34

1 Answer 1

Reset to default
7

The commands you're using can only extract data from the first entry in a ZIP archive; this is mentioned explicitly in the funzip manpage:

funzip without a file argument acts as a filter; that is, it assumes that a ZIP archive (or a gzip'd(1) file) is being piped into standard input, and it extracts the first member from the archive to stdout.

faredata2013.zip contains multiple entries, so you need to use unzip to extract them. If you want to extract them to stdout, you can use unzip with the -c option, and add -q if you just want the raw contents of all the files in the archive. (-c extracts the archive's contents to stdout, by default with a header giving each file's name before its contents; -q extracts without outputting the file names). You can also use the -p option instead of both -c and -q.

4
  • @Stepen - Thanks, I've read this funzip's docs part.. I've tried above command with -c and -cq args (for funzip) - still the same behavior. Did it work for you? ...Or did you mean these args for unzip - I thought that it doesn't support decompression to stdout
    – Oleg Shirokikh
    Jun 22, 2015 at 8:49
  • They're supposed to be used with unzip, I'll clarify that.
    – Stephen Kitt
    Jun 22, 2015 at 8:50
  • OK, understood.. unzip can write to stdout but can't read from it... Is it really the case that it's not possible to decompress piped multi-part files at all...
    – Oleg Shirokikh
    Jun 22, 2015 at 8:55
  • Yes, that's one of the limitations of unzip; it can extract to stdout but can't read an archive from stdin. jar can process archives from its stdin but can't output to stdout!
    – Stephen Kitt
    Jun 22, 2015 at 9:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .