1

I have written a main script that calls a system program tens of thousands of times. The main script creates and provides this program with an input file on each call. Upon completion, the program creates an output file which is read by the main script. The program is run with: program inputFile outputFile

The main script is running slowly, and I believe that this is due to the large I/0 overhead incurred whilst running program.

Is it possible to run this program without writing or reading to disk?

Note that both inputFile and outputFile may contain multiple lines. The internals of the program could be changed, but I would prefer to not do this. Unfortunately I couldn't find relevant advice elsewhere.

closed as unclear what you're asking by polym, maxschlepzig, Ramesh, Gilles, jordanm Aug 5 '14 at 23:38

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • can you modify the called system program? or is it a proprietary binary blob? please update your question with relevant information. – lesmana Aug 5 '14 at 20:38
  • 2
    Profile, don't speculate. Attempting to make something faster when you have no idea whether it's a bottleneck is a waste of time. If the file is small enough to fit in memory and short-lived, it wouldn't be written to disk anyway, it would stay in the cache. – Gilles Aug 5 '14 at 23:02
1

It may depend on what your script is doing with the output from the program.  If you are doing something complicated that requires processing the file multiple times, you may be out of luck.  (Let's call that "point zero".)

First, check to see whether the program can be told, directly and unambiguously, to write its output to the standard output instead of a named file.  Some programs interpret an argument of just a dash (-) to mean standard input or standard output, as appropriate.  So try

program inputFile - | your processing

Second, try the same with /dev/stdout as the output file name:

program inputFile /dev/stdout | your processing

Third, try a named pipe:

myFifo=$(mktemp)
program inputFile "$myFifo" &
your processing < "$myFifo"
rm "$myFifo"

Not the answer you're looking for? Browse other questions tagged or ask your own question.