I've hundred of text files I need to CURL from remote host.

Currently I've a loop which download each file and then compress it as new .zip entry.

Is it possible to directly pipe CURL's output to an archive utility (I don't care its zip, gzip, tar, rar or anything else...) in order to have each CURL's result as an archive entr?

| improve this question | | | | |

If curl’s standard output isn’t a terminal, it outputs the downloaded contents there instead of writing them to a file. You can then combine this with other tools’ ability to read from standard input. If you want to produce an archive containing multiple entries, you need to use a tool which can be told what name to use; for example, 7z:

curl https://raw.githubusercontent.com/akka/akka/master/README.md | 7z a -siREADME.md akka.7z
curl https://raw.githubusercontent.com/akka/akka/master/RELEASING.md | 7z a -siRELEASING.md akka.7z

will produce an akka.7z archive containing README.md and RELEASING.md:

$ 7z l akka.7z
   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
2019-03-06 10:15:45 .....         3236         1457  README.md
2019-03-06 10:16:18 .....         3001         1437  RELEASING.md
------------------- ----- ------------ ------------  ------------------------
2019-03-06 10:16:18               6237         2894  2 files

(7z doesn’t support this with all archive formats; for example, I couldn’t get it to work with ZIP files.)

| improve this answer | | | | |

Use the -o option (output file) for curl, specify the output file as - (a single dash, meaning 'stdout') and pipe it to anything you like.

From the curl man page:

-output <file>
          Write output to <file> instead of stdout. If you are using {} or [] to fetch multiple documents, you can use '#'
          followed by a number in the <file> specifier. That variable will be replaced with the current string for the URL
          being fetched. Like in:

            curl http://{one,two}.site.com -o "file_#1.txt"

          or use several variables like:

            curl http://{site,host}.host[1-5].com -o "#1_#2"

          You may use this option as many times as the number of URLs you have.

          See also the --create-dirs option to create the local directories dynamically. Specifying the output as  '-'  (a single dash) will force the output to be done to stdout.

Upon further reflection, I think I see the desired results: Each file downloaded should be saved in the chosen archive/compression format rather than a single archive containing all off the files from the downloaded batch. If I am wrong, and a single archive is the goal, the answer from Stephen Kitt seems like a better choice.

Based on what I believe to be the goal, I think you process can be improved somewhat, but not to the level you're hoping. Piping the output directly from curl will loose the file name. You do not, however, need to run it through a loop. Doing so negates curl's ability to re-use connections, avoiding multiple connection/handshake exchanges and increasing speed. The loop also makes curl pause between each download, while the compression takes place. I am going to presume that you can use the in-built expansion ability of curl and have some method in place to populate the same.

If you have a dedicated location for the downloads, which will be empty before the invocation of curl, you can use that (eliminating the first and last steps below). Otherwise you will need to create a temporary directory for the downloads. If that is on the same disk partition as their final destination, the "move" will be simple and quick.

The process, once you have the file list created, is to:

  • Create the temporary download directory
  • Invoke curl one time with the complete list of files included
  • Direct curl to save the files, properly named, in the download location
  • Invoke find on the download location
  • Use the -exec option of find to archive all the downloaded files
  • Move the archived files into their storage location.

A single command line will do the job:

mkdir -p temp_down && 
pushd temp_down >/dev/null && 
curl "http://www.arowtemple.com/{index,about,contact,directors,covens,temple,lessons,priesthood}.html" -o "#1.html" &&
find . -type f -exec sh -c 'zip -rms9T --move "$0.zip" "$0"' {} \; &&
popd >/dev/null

Of note are the use of quotes in the 3rd and 4th lines. The first set of double quotes in line 3 allow Bash to expand the variable, if needed, with the list of files to get, while preventing Bash from expanding the brace contents. The second set keeps the created file names shell-safe. With curl expanding the brace contents the '#1' near the end will be replaced with the file name of each file retrieved. The single quotes in line 4 keeps the command intact when it is passed to the sub-shell, and the double quotes keep the file name shell-safe. The $0 entries are not a typo, they are not supposed to be $1 as would be expected.

If your downloads are supposed to all be collected into a single directory, you can remove the --create-dirs option from the curl command, and if you wish to keep the original files along with the archived versions, remove the --move option from the find-zip command. Of course, the zip command can be replaced with whichever archive/compression program you choose.

| improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.