Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I often want to feed relatively short string data (could be several lines though) to commandline programs which accept only input from files (e.g. wdiff) in a repeated fashion. Sure I can create one or more temporary files, save the string there and run the command with the file name as parameter. But it looks to me as if this procedure would be highly inefficient if data is actually written to the disk and also it could harm the disk more than necessary if I repeat this procedure many times, e.g. if I want to feed single lines of long text files to wdiff. Is there a recommended way to circumvent this, say by using pseudo files such as pipes to store the data temporarily without actually writing it to the disk (or writing it only if it exceeds a critical length). Note that wdiff takes two arguments and, as far as I understand it will not be possible to feed the data doing something like wdiff <"text".

share|improve this question
    
Can this be solved via xargs? –  N.N. Feb 6 '13 at 10:47
    
Don't know, but it would not be obvious to me how. As far as I understand xargs would make the input lines from the file string arguments for the command. But I need the opposite. –  highsciguy Feb 6 '13 at 11:07
    
@rahmu I had a look, but I think the problem setting is a little different there. At least I don't see how the answers would help. The accepted answer to produce temporary files properly is essentially what I wan't to avoid, if not there is some kind of buffering which actually prevents writing the files. I have limited understanding of how the temp files work! –  highsciguy Feb 6 '13 at 11:12
    
What's wrong with echo $data_are_here | dumb_program? –  vonbrand Feb 6 '13 at 12:13
1  
This would support only one input file and not all programs would read from stdin. –  highsciguy Feb 6 '13 at 12:42

2 Answers 2

up vote 4 down vote accepted

Use a named pipe. By way of illustration:

mkfifo fifo
echo -e "hello world\nnext line\nline 3" > fifo

The -e tells echo to properly interpret the newline escape (\n). This will block, ie, your shell will hang until something reads the data from the pipe.

Open another shell somewhere and in the same directory:

cat fifo

You'll read the echo, which will release the other shell. Although the pipe exists as a file node on disk, the data which passes through it does not; it all takes place in memory. You can background (&) the echo.

The pipe has a 64k buffer (on linux) and, like a socket, will block the writer when full, so you will not lose data as long as you do not prematurely kill the writer.

share|improve this answer
    
Ok, thanks, this works also with two named pipes and wdiff. But I thought to understand that there is a certain (small) amount of memory available for the pipe as buffer. What happens if I exceed the buffer size? –  highsciguy Feb 6 '13 at 11:57
    
I added a final paragraph about that issue. –  goldilocks Feb 6 '13 at 12:19
1  
/tmp is configure in most distros to use a tmpfs filesystem which is in RAM. When you write a file in /tmp it goes directly to your RAM which makes is a good answer for semi-resilient files that have to be accessed quickly and rewritten many times. –  Gael Feb 7 '13 at 9:57

In Bash, you can use the command1 <( command0 ) redirection syntax, which redirects command0's stdout and passes it to a command1 that takes a filename as a command-line argument. This is called process substitution.

Some programs that take filename command-line arguments actually need a real random-access file, so this technique won't work for those. However, it works fine with wdiff:

user@host:/path$ wdiff <( echo hello; echo hello1 ) <( echo hello; echo hello2 )
hello
[-hello1-]
{+hello2+}

In the background, this creates a FIFO, pipes the command inside the <( ) to the FIFO, and passes the FIFO's file descriptor as an argument. To see what's going on, try using it with echo to print the argument without doing anything with it:

user@host:/path$ echo <( echo hello )
/dev/fd/63

Creating a named pipe is more flexible (if you want to write complicated redirection logic using multiple processes), but for many purposes this is enough, and is obviously easier to use.

There's also the >( ) syntax for when you want to use it as output, e.g.

$ someprogram --logfile >( gzip > out.log.gz )

See also the Bash redirections cheat sheet for related techniques.

share|improve this answer
    
This is not supported in KSH –  chanchal1987 Jul 3 at 9:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.