3

I would like to automatically create a compressed archive from a file given via stdin (or create the archive, if it doesn't exist yet), while specifying the file name that it's going to have inside the archive.

7zip seems to have a feature that does exactly what I'm after (doku).

The usage is like this:

cat some_file.png | 7z a archive.7z -t7z -si"custom_filename1.png"

This adds a file called "custom_filename1.png" to archive.7z. However, I would like to do this without 7zip if possible, because I think 7zip would not be preinstalled on most Linux distributions. I searched for a way to do this with tar, but only found this post which doesn't seem to work on my machine.

I don't want to create temporary files.

5
  • Why would you need to create a tar archive? Would it not be eough to just compress it with e.g. gzip?
    – Kusalananda
    May 5, 2019 at 11:40
  • I need to put multiple files into the archive which is not possible with gzip, but with tar and 7zip it is.
    – lug_wrench
    May 5, 2019 at 11:56
  • But if the data is coming in over the standard input stream, how would you distinguish the individual files? Or are you adding a file at a time by running the pipeline multiple times? What's the actual command at the start of your pipeline?
    – Kusalananda
    May 5, 2019 at 12:02
  • 2
    The actual command is a python script which uses convert and outputs a png to stdout. Yes, I wish to run the command several times, each time specifying a different filename.
    – lug_wrench
    May 5, 2019 at 12:32
  • How about just writing a shell script which wraps around tar and appends to the tar file? Or create a temporary file yourself. tar cannot append from stdin.
    – Ned64
    May 5, 2019 at 15:09

5 Answers 5

3

What you seem to want to do is to add files to a tar archive, one by one, from the standard output of some other command.

Assuming that the archive exists, you may add a given file to it using

tar -u -f archive.tar filename

or, if you are working with compressed archives,

tar -uz -f archive.tar.gz filename

The file called filename must exist in the filesystem. This means that your workflow would have to save the file and then add it to the archive before deleting the file again.

Possibly something like this:

somecommand filename.png >newfile1.png &&
tar -u -f archive.tar newfile1.png &&
rm newfile1.png

The && between the commands would ensure that a command does not run if the previous command failed for whatever reason.

Note that you would have to add each file with a new name into the archive, as adding a file with the same name as a pre-existing file would "hide" the older file in the archive.

3
  • 1
    Like I specified, I don't want to create a temporary file.
    – lug_wrench
    May 5, 2019 at 16:45
  • @lug_wrench You may have to create a file. Are you concerned about the disk access for speed reasons? Or is the file too large? In the first case why not mount a temporary file system to save the temp file without disk access (as long as the RAM suffices).
    – Ned64
    May 5, 2019 at 17:11
  • @lug_wrench If you're wanting to add things to a tar archive, then you will need te create a file for tar to add.
    – Kusalananda
    May 5, 2019 at 19:14
3

Using zip and bash:

cat some_file.png | zip -v --fifo archive.zip /dev/stdin
printf '%s\n' '@ dev/stdin' '@=custom_filename1.png' | zipnote -w archive.zip

# view archive contents:
unzip -l archive.zip

Archive:  archive.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
     5200  2020-09-26 12:41   custom_filename1.png
---------                     -------
     5200                     1 file

You mentioned tar, but you describe two constraints:

  1. create an archive by reading from a pipe - without creating a temp file
  2. use a widely-available archive tool

Your "no temp file" constraint means that you must create the archive from a pipe. Tar only reads files - not pipes - so tar is probably a no-go.

zip --fifo reads from a pipe and saves that output as a file in an archive. Zip may be less prevalent than tar, but certainly is more prevalent than 7z.

zip reads the file contents from /dev/stdin:

cat some_file.png | zip -v --fifo archive.zip /dev/stdin
    zip warning: Reading FIFO (Named Pipe): /dev/stdin
  adding: dev/stdin (in=39032) (out=1468) (deflated 96%)
total bytes=39032, compressed=1468 -> 96% savings

This saves the contents of some_file.png to archive.zip as file dev/stdin

Then rename the saved file to custom_filename1.png with zipnote. This rename action is atomic - no tempfiles are created.

2
  • 1
    And, even if you needed to extract the name of the added pseudo-file from the zip verbose listing, you wouldn’t need to use both grep and awk.   Change grep adding: | awk '{print $2}' to awk '/adding:/ {print $2}'. Sep 26, 2020 at 20:55
  • Updated the solution with @StéphaneChazelas improvements
    – yolabingo
    Sep 26, 2020 at 23:55
1

Pack up a file by filename with the output as filename.tgz

tar czf filename.tgz filename

Pack up one or more from a list of file(s)

tar czf files.tar.gz --files-from ./filelist

Same thing but by different method

ls -1b . >./filelist && tar czf ./files.tar.gz --files-from ./filelist

Add (update tarball) with additional file(s)

tar czuf ./files.tar.gz ./newfile

note: depending on your version of tar options may or may not require a preceding hyphen ( - )

options legend

  • c create
  • z use gzip compression
  • u update (add additional file(s))
  • f tar file name
  • --files-from take contents of file for list of files to pack/added

See also the tar man page

HTH

1

Given that you're just compressing one file, maybe you want gzip:

cat somefile.png | gzip > somefile.png.gz

Tar relies on actual filenames / directories, and so might be less suited for single-file direct use in a pipe like this. Most environments have gzip. Then:

gzip -d somefile.png.gz
0

You could easily bypass that by going to file manager and right clicking on your file and clicking compress and then make it whatever filetype you want.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .