7

Is there a shorter way of writing this? Basically output a command to a file, then use the file as input for the next command. I also want to keep the file to view afterwards.

cmd1 > verylong.txt; cmd2 < verylong.txt

I know I can do

cmd1 | tee verylong.txt | cmd2

But since I expect "verylong.txt" to be a huge file, I thought it would be less efficient to use pipe since that would hold the entire file in memory. Whereas if I use file input then it would process it one line at a time. (Or is my assumption wrong?)

It would be great if I could do something elegant like

cmd1 > verylong.txt > cmd2

15

As far as I know, cmd1 | tee verylong.txt | cmd2 will not hold the whole file in memory. In fact, if cmd2 was to wait too long before consuming its input, cmd1 might block on a write call and unblock only when cmd2 starts reading again.

The reason for that is that there is a buffer for the pipe, and that buffer, by default, is limited to a certain reasonable size.

Of course, the story might be different if cmd2 is sort (or something alike) where the entire input must be read before the command is able to write its output. In that case, the entire file content might be held in cmd2 memory, but that is independent of whether a pipe or an intermediary file was used for the input of that command.

  • 3
    sort doesn't store the whole file in memory, it's got a buffer with a maximum size as well and resorts to temporary files when that maximum is reached. – Stéphane Chazelas Nov 21 '14 at 15:26
  • @StéphaneChazelas Good to know, have an upvote! ;) I'll update the answer to be less assertive in that "hold into memory" part. – user43791 Nov 21 '14 at 16:03
6

The already given answer is correct. But if your goal is to selectively read your verylongfile.txt w/ cmd2, sed might be another option.

cmd1 | sed -e 'w verylongfile.txt' -e '/notinteresting/d' | cmd2

sed will write all of its input to the outfile, but only the bits that do not match the /notinteresting/ address to the pipe. Or you might negate the action with /interesting/!d which would write only the lines that match the interesting address to the pipe.

If this is not your goal, use tee instead, though - it is a more efficient tool for writing the whole of its input to both the outfile and the pipe.

0

There is a clever trick with tee and subshells:

cat source.lst | tee >(doSomething.sh) >(somethingElse.sh) | somethingFinal.sh

I've done this before

pv -perl source.list | tee >(doSomething.sh) >(somethingElse.sh) | md5sum

pv will give you a progress bar, an ETA, and a running line total. Then source.lst will be fed to doSomething.sh and somethingElse.sh (and on different CPUs!) Finally we'll get an md5sum of that hugefile, just for academic purposes.

-5

Whats wrong with simple two line batch file? Like:

Cmd1 >filespec
Cmd2 <filespec

Or

cmd1 >filespec
cmd2 filespec

either way, the file is left in mass storage.

  • For some reason the site is not letting me key in a less than symbol. and dropping the second part of cmd2. So in words cmd1 redirect out to file. next line cmd2 redirect in from file. OR for cmd2 , just put the filename as the first parameter and cmd2 just opens the file. – user92319 Nov 21 '14 at 4:39
  • You use &lt; for the < symbol... – jasonwryan Nov 21 '14 at 4:57
  • 1
    One difference between cmd | tee file | cmd and cmd >file; cmd <file is that the commands in the first group are executed in parallel - which is to say that they all start at the same time. And so cmd2 can process cmd1s output as it is written, whereas cmd1; cmd2 are two commands that are executed one after the other - or, in other words, cmd2 must wait on cmd1 to complete before processing anything. – mikeserv Nov 21 '14 at 4:58
  • The author of the question specifically asked for a command that's different from that because the files are very large. Executing the files using your methodology would be a lot slower than cmd1|tee file|cmd2. – Mark D Nov 26 '14 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.