3

Suppose I do a grep that gives me a lot of results. I want to use the output of grep and pass it to a command that takes a filename as it's first argument. For example -

myCommand filename.txt

I would like to do something like this -

grep "xyz" | myCommand

However, since myCommand doesn't read from stdin, this piping does not work. I don't want to create a temporary file on disk with the results of grep. Is it possible by any other means to create a "file", say in memory and let myCommand use that?

  • Possibly related. I asked a similar question here once. It's okay to create the temporary files. You'll find some good practice on how to do this in the answers linked. – rahmu Jul 24 '13 at 14:14
5

A simple solution is to use Bash's support for process substitution, like this:

myCommand <(grep "xyz" somefile)

This will connect the output of the grep command to a file descriptor and then pass /dev/fd/NNN (where NNN is the fd number) to your command as the input file (for example, your command might actually get run as myCommand /dev/fd/63). This solution does not create a temporary file.

You can also create a named pipe and use that to transfer data around:

$ mkfifo /tmp/myfifo
$ grep "xyz somefile > /tmp/myfifo &
$ myCommand /tmp/myfifo

But really, the first solution is simpler.

Stephane Chazelas comments:

zsh also has myCommand =(grep xyz somefile) which uses a regular temporary file (for cases where myCommand can't accept a pipe).

This is important because some commands expect to be able to perform random access on their input file, which won't work with a pipe. Using this syntax you get the benefits of temporary files without needing to worry about cleaning them up yourself.

  • Note that although process substitution is great here, it's not supported by all shells or on all operating systems. The fifo approach should work just about everywhere. – Mat Jul 24 '13 at 14:19
  • myCommand <(grep "xyz" somefile) didn't work on Ubuntu for me. Now I'm trying the second approach. – CodeBlue Jul 24 '13 at 14:28
  • @CodeBlue, note that the first solution does, as I indicated, require Bash. If you're using a different shell there might be a different syntax (or there might not be an equivalent feature). – larsks Jul 24 '13 at 14:31
  • 2
    process substitution originated in ksh93 and is also available in zsh and bash. zsh also has myCommand =(grep xyz somefile) which uses a regular temporary file (for cases where myCommand can't accept a pipe). – Stéphane Chazelas Jul 24 '13 at 14:33
  • Stephane: good to know, I've added this to the answer. – larsks Jul 24 '13 at 14:37
2

Most distros now include a RAM disk for this exact purpose. You can write files to it like normal, it's typically called /dev/shm. You can check if you have it like so:

$ mount |grep shm
tmpfs on /dev/shm type tmpfs (rw)

If you have it you can write to it just like any other filesystem:

$ echo "hello world" > /dev/shm/somefile.txt

And see it using normal tools:

$ ls -l /dev/shm/
total 680
-r-------- 1 saml saml 67108904 Jul 22 20:55 pulse-shm-3981461026
-r-------- 1 saml saml 67108904 Jul 18 18:06 pulse-shm-799251885
-rw-rw-r-- 1 saml saml       12 Jul 24 10:59 somefile.txt

However this consumes no actual diskspace, it's using RAM.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.