3

Suppose I do a grep that gives me a lot of results. I want to use the output of grep and pass it to a command that takes a filename as it's first argument. For example -

myCommand filename.txt

I would like to do something like this -

grep "xyz" | myCommand

However, since myCommand doesn't read from stdin, this piping does not work. I don't want to create a temporary file on disk with the results of grep. Is it possible by any other means to create a "file", say in memory and let myCommand use that?

1
  • Possibly related. I asked a similar question here once. It's okay to create the temporary files. You'll find some good practice on how to do this in the answers linked.
    – rahmu
    Commented Jul 24, 2013 at 14:14

2 Answers 2

6

A simple solution is to use Bash's support for process substitution, like this:

myCommand <(grep "xyz" somefile)

This will connect the output of the grep command to a file descriptor and then pass /dev/fd/NNN (where NNN is the fd number) to your command as the input file (for example, your command might actually get run as myCommand /dev/fd/63). This solution does not create a temporary file.

You can also create a named pipe and use that to transfer data around:

$ mkfifo /tmp/myfifo
$ grep "xyz somefile > /tmp/myfifo &
$ myCommand /tmp/myfifo

But really, the first solution is simpler.

Stephane Chazelas comments:

zsh also has myCommand =(grep xyz somefile) which uses a regular temporary file (for cases where myCommand can't accept a pipe).

This is important because some commands expect to be able to perform random access on their input file, which won't work with a pipe. Using this syntax you get the benefits of temporary files without needing to worry about cleaning them up yourself.

7
  • Note that although process substitution is great here, it's not supported by all shells or on all operating systems. The fifo approach should work just about everywhere.
    – Mat
    Commented Jul 24, 2013 at 14:19
  • myCommand <(grep "xyz" somefile) didn't work on Ubuntu for me. Now I'm trying the second approach.
    – CodeBlue
    Commented Jul 24, 2013 at 14:28
  • @CodeBlue, note that the first solution does, as I indicated, require Bash. If you're using a different shell there might be a different syntax (or there might not be an equivalent feature).
    – larsks
    Commented Jul 24, 2013 at 14:31
  • 2
    process substitution originated in ksh93 and is also available in zsh and bash. zsh also has myCommand =(grep xyz somefile) which uses a regular temporary file (for cases where myCommand can't accept a pipe). Commented Jul 24, 2013 at 14:33
  • Stephane: good to know, I've added this to the answer.
    – larsks
    Commented Jul 24, 2013 at 14:37
2

Most distros now include a RAM disk for this exact purpose. You can write files to it like normal, it's typically called /dev/shm. You can check if you have it like so:

$ mount |grep shm
tmpfs on /dev/shm type tmpfs (rw)

If you have it you can write to it just like any other filesystem:

$ echo "hello world" > /dev/shm/somefile.txt

And see it using normal tools:

$ ls -l /dev/shm/
total 680
-r-------- 1 saml saml 67108904 Jul 22 20:55 pulse-shm-3981461026
-r-------- 1 saml saml 67108904 Jul 18 18:06 pulse-shm-799251885
-rw-rw-r-- 1 saml saml       12 Jul 24 10:59 somefile.txt

However this consumes no actual diskspace, it's using RAM.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .