Linked Questions

395
votes
13answers
202k views

Turn off buffering in pipe

I have a script which calls two commands: long_running_command | print_progress The long_running_command prints a progress but I'm unhappy with it. I'm using print_progress to make it more nice (...
89
votes
4answers
20k views

In what order do piped commands run?

I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about ...
73
votes
7answers
38k views

When would you use an additional file descriptor?

I know you can create a file descriptor and redirect output to it. e.g. exec 3<> /tmp/foo # open fd 3. echo a >&3 # write to it exec 3>&- # close fd 3. But you can do the same ...
23
votes
5answers
5k views

On-the-fly stream compression that doesn't spill over into hardware resources?

I have 200 GB free disk space, 16 GB of RAM (of which ~1 GB is occupied by the desktop and kernel) and 6 GB of swap. I have a 240 GB external SSD, with 70 GB used1 and the rest free, which I need to ...
11
votes
3answers
2k views

How to copy a directory which root can't access to a directory that only root can access?

I have a directory on an nfs mount, which on the server is at /home/myname/.rubies Root cannot access this directory: [mitchell.usher@server ~]$ stat /home/mitchell.usher/.rubies File: `/home/...
16
votes
2answers
1k views

Understanding piped commands in Unix/Linux

I have two simple programs: A and B. A would run first, then B gets the “stdout” of A and uses it as its “stdin”. Assume I am using a GNU/Linux operating system and the simplest possible way to do ...
7
votes
4answers
3k views

output to file, then use file for input

Is there a shorter way of writing this? Basically output a command to a file, then use the file as input for the next command. I also want to keep the file to view afterwards. cmd1 > verylong.txt; ...
3
votes
4answers
5k views

Compare awk vs. grep

I have run below two commands for a very large files grep -E 'string1|string2' 151103*.log|grep 'string3' | grep string4 awk '/string1|string2/ && /string3/ && /string4/' 151103*.log ...
13
votes
1answer
3k views

I'm confused as to why “| true” in a makefile has the same effect as “|| true”

I have a makefile where I'm stopping a service before removing a file. When it couldn't stop the service, it would break on error. This is clearly unwanted so I thought I'd add || true but missed a |. ...
3
votes
3answers
4k views

How does Bash pipe large amounts of data?

Let's say you want to cat the contents of a really big file, but want to view it a few bits at a time. Let's say one were to do the following: $ cat /dev/sda1 | less As a programmer of languages ...
3
votes
5answers
2k views

How to run a pipe safely and sequentially?

In Linux, is it possible to run a pipe: cmd1 | cmd2 in such a way that: cmd2 doesn't start running until cmd1 has completely finished, and If cmd1 has an error, cmd2 doesn't run at all and the exit ...
6
votes
1answer
1k views

Can writing to stdout place backpressure on a process?

In stream processing and queuing, we have this notion of backpressure, which is that if a producer process is going faster than a consumer process, we should have a mechanism for slowing down the ...
4
votes
3answers
107 views

In what format does piped output get sent and received from one command/program to another? [duplicate]

When a command is piped to another command in what way or in what format does the piped output exist/get sent? Is it a temporary file? Is it a string? And how does the command that receives the piped ...
2
votes
3answers
503 views

Replacing sshfs with ssh

I love to do just this: $ sshfs mountPoint myServer $ cp thisFile mountPoint I am now using LiveCD and I do not have sshfs utility available and I need run $ sudo dd /dev/sdb2 > mountPoint, how ...
1
vote
1answer
625 views

Anonymous Pipe Kernel Buffer Size

I am trying to understand the structures associated with a call to the pipe syscall. int pipe(int pipefd[2]); From my understanding, there will be a "read" and "write" kernel buffer/structure ...

15 30 50 per page