2

I am trying to create a one-line Linux command to combine two PDF files - which are downloaded from a URL - using Ghostscript. However, I do not want to create any temporary files (everything should be done in memory).

The following command does not appear to work (I tried achieving this by process substitution).

gs -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=combined.pdf <(curl http://example.com/one.pdf) <(curl http://example.com/two.pdf)

When I run this command, it gives me the error below.

**** Warning:  An error occurred while reading an XREF table.
**** The file has been damaged.  This may have been caused
**** by a problem while converting or transfering the file.
**** Ghostscript will attempt to recover the data.
Error: /ioerror in --run--
Current allocation mode is local
Last OS error: Illegal seek
GPL Ghostscript 9.18: Unrecoverable error, exit code 1

I believe what is happening is that the Ghostscript command is being run before the two input PDFs have had a chance to finish downloading, perhaps there is a way to wait for this to happen.

1 Answer 1

5

This is not the answer you're looking for, but it is a practical alternative.


First, some background:

All pipe types have the downside that the reader cannot seek in them, including the ones Bash uses for <(command-list). I do not know if GhostScript needs to seek in input files, or whether it just reads the entire files in memory, but in general, for many file formats, pipe-based input may be slower or use more memory (due to no-seek workarounds) than file inputs.

(I am not sure, because I lean towards robustness. I want my scripts to work (give me reliable results) first, and be lightweight and fast second. For the reasons listed below, avoiding temporary files gives me no speedup or lesser resource usage on systems I use.)

In many (most?) Linux distributions, Solaris (since SunOS 4, and Solaris 2.1), NetBSD (4.0 and later), FreeBSD (7.0 and later), DragonFly BSD, and OpenBSD (5.5 and later), /tmp is usually a tmpfs, a RAM-based filesystem. On these systems, avoiding temporary files is then counterproductive (except for the cases where you absolutely know the application(s) you feed it to can process the input as a stream, i.e. via a pipe, without downsides).

Typical counterexamples (i.e. cases when you do use a pipe instead of temporary files) are simple filtering or processing via e.g. sed or awk. GhostScript is definitely not "a simple filtering or processing program".

All current operating systems are smart enough to keep recently used files in memory, rather than writing and reading them from the storage. (They will typically save the contents to the disk once, but not read back, in this kind of temporary file situations.) So, even when /tmp is not a RAM-based filesystem, temporary files are kept in memory, and at most just written once to the disk. It means, even in the non-RAM-based /tmp, temporary files are a good option.

All together, the above means that in normal use cases, you do not want to avoid temporary files, especially when they are fed to converters or applications for processing.

Indeed, usually the reason behind wishing to avoid temporary files is some variant of

I don't want my scripts to leave behind unnecessary temporary files if I interrupt them, or they fail/exit with an error/do not work right.

With Bash and mktemp utility, that is trivial to avoid. I've used the following idiom for several years now:

#!/bin/bash
Work=$(mktemp -d) || exit 1
trap "cd / ; rm -rf '$Work'" EXIT

This creates a temporary directory (under /tmp/), which will be automatically removed when the script exits. (The Linux coreutils mktemp makes the directory accessible only to the owner user; no access to group or others, so this is quite safe, too.)

The trap bash built-in command is formulated (via these particular quotes exactly thus) so that even if you happen to change the Work environment variable later on, the correct (original) temporary directory will be removed, because the variable is expanded when the trap is set and not when the trap triggers.

After the above, you can use e.g.

curl 'http://www.example.com/one.pdf' > "$Work/one.pdf" || exit 1
curl 'http://www.example.com/two.pdf' > "$Work/two.pdf" || exit 1

gs -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=combined.pdf "$Work/one.pdf" "$Work/two.pdf" || exit 1

There is no need to add any cleanup after this, as the exit trap will handle that automatically -- even if you interrupt the script using Ctrl+C or other signal.

2
  • Thanks for such a detailed answer, I'm sure that it will help others as well! Whilst yours is not quite the solution I was expecting, I appreciate that in many cases piping will not always have such a benefit.
    – Zak
    Oct 17, 2016 at 19:09
  • @Zak: I only now noticed the Last OS error: Illegal seek line in the error you showed. That error is indeed GhostScript telling that it cannot read PDF input files from pipes, as it needs to seek (read from different parts) in the PDF input file. Apologies; I blame .. uh .. this being Monday. Ahem. (This does mean you cannot avoid temporary files if you are processing multiple PDF files with GhostScript.) Oct 17, 2016 at 19:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .