Questions tagged [pipe]
A Unix pipe connects file descriptors of two processes. A pipe is created with the POSIX pipe() function declared in <unistd.h>. Shells provide pipe creation between processes using "|".
1,822
questions
3
votes
2
answers
322
views
Piping and input stream redirection to same command, which input does command take?
I was trying to run the command given below
cat numbers.txt | sort < words.txt
The above command has a pipe from cat command feeding to input stream of sort command. But at the same time, input ...
- 133
0
votes
1
answer
20
views
Buffering follow + automatic pagination of output
I have a utility that continuously outputs live logs from a program. But it can be quite hard to read the logs lines as they scroll past me in the terminal. Ideally it should scroll to one full page, ...
- 101
0
votes
0
answers
26
views
Bash equivalent of zsh alias at end of command (for pipes) [duplicate]
In zsh, I can set an alias at the end of a command, e.g. making G pipe through grep or L pipe through less.
Can I do something similar in bash? E.g. ls G key would expand to ls | grep key or ls L ...
- 111
0
votes
1
answer
70
views
When simply `| cat -`, I get file descriptor errors
Here's the working diff command:
$ diff -u <(echo 'foo:bar:baz' | tr : "\n") <(echo 'foo:baz' | tr : "\n")
--- /dev/fd/11 2023-08-30 13:11:50
+++ /dev/fd/13 2023-08-30 13:...
- 101
0
votes
1
answer
53
views
Continuously polling a named (FIFO) pipe
I am trying to use a named pipe to monitor the activity of a running shell script. That shell script will write to the pipe that it has "Started" processing. When it has completed ...
- 1
2
votes
0
answers
41
views
Waiting until an application is reading (over a TTY)
I'm writing an expect-style program to interact with a command-line application on Linux. Is there any way that my program can know when the application on the other side of the PTY tries to perform a ...
- 21
9
votes
1
answer
581
views
Exit status of pipe with timeout command behaves differently in interactive shell vs shell script
Interactive sh shell session:
$ sh
$ timeout 1 yes | sed -n s/a/b/p ; echo $?
Terminated
143
$
Non-interactive script via sh -c:
$ sh -c 'timeout 1 yes | sed -n s/a/b/p ; echo $?'
0
$
Why do those ...
- 2,980
-1
votes
2
answers
156
views
Difference in bash 3 between pipe and here-string
If I run:
IFS=':' cat <<< $(echo $PATH)
then the ":" are used for word splitting and the output contains spaces instead of ":", but if I run:
echo $PATH | IFS=':' cat
...
- 206
2
votes
1
answer
147
views
What are the security implications of piping apt output compared to apt-get?
I tried to install all python3 packages so that I don't have to manually go to terminal and install one everytime. However due to some dependency issues apt refused to proceed:
sonic@boomboom:~$ sudo ...
- 21
0
votes
1
answer
37
views
Pipe : gunzip files and concatenate
I feel like this should be simple. I have a directory with >1000 gzip-ed files. I'd like to gunzip them all and concatenate.
Running gzip -dk *.gz | cat > output.file produces the following ...
- 169
0
votes
1
answer
67
views
Continuously display last line of command output without clearing the screen
I want to print only the last line of the output of a command on the same line continuously without clearing the screen. This would be part of another function. Let's say my script has the following ...
- 1
1
vote
1
answer
53
views
Creating PKCS#12 container from input from stdin
I want to create a PKCS#12 (a.k.a PFX) container for a private key and a certificate using openssl.
My key and cert are contained in a single (unecnrypted) file:
$ cat d.pem
-----BEGIN CERTIFICATE-----...
- 290
0
votes
0
answers
26
views
how does data flows through pipe in linux when sorting?
suppose I do something like
cat afile [ | possibly awk or similar to process the file] | sort
I know data flows something like
afile -- stdout -- os buffer -- stdin -- sort
When the size of afile is ...
- 101
0
votes
0
answers
24
views
Capture stderr of program into a variable without affecting stdout and interactive features [duplicate]
I have a program that asks the user for input and provides interactive features such as moving the cursor, deleting text, and more. After the user completes entering the input, the program clears the ...
- 176
0
votes
0
answers
33
views
create a compressed tar archive from files extracted from another tar archive [duplicate]
I'm trying to do what I specified in the object. I have a tar.gz archive (i.e. archive.tar.gz) and I want to:
extract only some file (e.g. *.csv) contained in a specific folder of the archive (e.g. ...
- 1
0
votes
1
answer
74
views
Piping output to tee breaks the conditional
I am trying to test for the presence of custom firewall-rules on servers. I am doing this by processing the iptables output with awk and returning an appropriate exit-code. This works as intended, and ...
0
votes
1
answer
53
views
bash: clarification on order of redirection and pipes
In an attempt to further understand bash, I am trying to replicate some functionalities of it. Right now I am in the process to implementing the redirecting and pipeline features. To further ...
1
vote
2
answers
58
views
Is there a program that concatenates non-seekable streams (size not known in advance) and can separate them again?
I'm trying to concatenate multiple input files/streams into one stream (using the imaginary command stream-cat), pipe that stream into ssh and on the remote host separate it back into individual files/...
- 133
0
votes
1
answer
121
views
How to pipe STDIO from a thread process to /dev/null?
I am trying to run Plarium Play with wine, but have encountered an odd issue. When trying to launch it from a regular desktop entry, I get this JavaScript error:
This does not happen if I launch from ...
- 95
3
votes
4
answers
340
views
Apply arithmetic into piped command
So far I have this:
sudo find /path/to/dir -type f |
xargs -d "\n" sudo stat -c "%Y %n" |
{arithmetic to check if %Y is between 1685518962 and 1685624474??} |
{show ...
- 143
1
vote
1
answer
48
views
Allow a sudo sub process to update Zenity running as original user
A script run by user 'staffer' runs zenity --progress.
It then calls sudo -u adminBod adminScript (and the STDOUT and STDERR are collected to logger)
I want adminScript which is running as adminBod to ...
- 2,819
3
votes
1
answer
86
views
How to ensure that SSH client leave stdout file descriptor clean for reuse?
This code terminates with error :
(
ssh localhost seq 100000
seq 100000
) | wc
#-> seq: write error: Resource temporarily unavailable
This is a minimal code to reproduce the write error.
...
- 814
1
vote
0
answers
34
views
How to pass a character device to something expecting a named pipe
Is it possible to convert a special file corresponding to a character device into a pipe?
Specifically if I had an embedded device connected to a linux system via USB, and the device could send/...
- 111
0
votes
0
answers
81
views
Performance limits of file I/O vs Unix pipes on multicore systems
I am running some benchmarks where I have a program that performs some file IO. It creates a fresh directory, writes and reads some files to that directory, and then deletes it. When I run this ...
- 131
1
vote
1
answer
52
views
How to pipe terminal interaction with running process over network?
How can I run a program that expects to be executing in a terminal in such a way that it's STDIO can be arbitrarily connected-to remotely?
Specifically, I want to be able to launch gdb on one host and ...
- 111
5
votes
4
answers
644
views
Copy the n-largest files from a certain directory to the current one
I am trying to copy five largest files from a certain directory to my pwd. Using cp specific/directory$(ls -S specific/directory | head -n) ./ copies the first file and then proceeds to produce cannot ...
- 257
0
votes
2
answers
156
views
How to pipe an object file into GCC for linking?
My custom compiler for a toy language uses Cranelift to generate object code, that I can write into a file main.o. To make this object file into an executable file, I invoke GCC with gcc -o main main....
- 103
0
votes
2
answers
243
views
Get exit status of the first command in a pipe in the second the command
My question is similar to Get exit status of process that's piped to another, but I want to get the exit status of the second command in the pipe within the pipe.
Here's an example -
false | echo $...
- 103
6
votes
1
answer
233
views
Download a file and piping to uncompress is not as fast as expected
I need to download and decompress a file as quickly as possible in a very latency sensitive environment with limited resources (A VM with 1 cpu, 2 cores, 128MB RAM)
Naturally, I tried to pipe the ...
- 113
0
votes
1
answer
149
views
problem with piplining using jq [duplicate]
The exact question was asked here:
Using jq within pipe chain produces no output
but the answser provided there did not solve my problem. Also, I read the following:
Turn off buffering in pipe
https:...
- 163
0
votes
2
answers
89
views
How is output passed as input in pipes?
I am trying to understand how pipes | work; in particular, in what way the output from the left command is passed as the input to the right command. Is the output from the left command treated as a ...
2
votes
2
answers
253
views
Can you compile a file without writing to disk?
I'd like to compile some code from within a C/C++ program. Here's how it would work. I would have a buffer in memory (dynamically allocated) that contains ASCII text that constitutes C code. From my C/...
- 213
0
votes
0
answers
66
views
`yes | head` why command yes here stops when head is over [duplicate]
The command yes prints "y" on stdout forever. Head prints the first 10 lines of the input.
Why yes stops when head stops ? I have my little idea but I'm just not sure so I ask here. To me ...
- 1
2
votes
2
answers
113
views
Store file outputs of command in bash variables
Suppose I have a utility foo that writes to n different files simultaneously. Different data is written to every file. What is written to the files is not known in advance. For example, the utility ...
- 101
0
votes
3
answers
278
views
Why does < exist to redirect to stdin? [closed]
Question
Why do shells implement alternative means like <<<, < <(command) and < /dev/fd/* to redirect something to stdin when pipes do exist?
Example
The | way (classic pipes)
echo '...
- 337
0
votes
0
answers
228
views
How to read an arbitrary process's stdout pipe in Linux?
I have a child process launched via Supervisor that's somehow become disconnnected from its parent and is running but not outputting to the parent's log file anymore.
However, it's still running, and ...
- 1,589
1
vote
2
answers
314
views
How to apply a patch as part of a pipe? In other words, how to patch stdin?
I want to apply a unified diff from mypatch.diff to stdin and output the result to stdout.
So far, I have tried:
patch -i mypatch.diff -o - -u originalfile
Which successfully applies mypatch.diff and ...
- 2,980
0
votes
1
answer
145
views
Getting first line of piped output sets exist status 141 which fails bash script with set -eo pipefail
I have a bash script that runs a command from which I want to get only the first line of output.
The obvious answer is some_command | head -1, right? Well, this works 99% of the time, but when the ...
- 446
7
votes
6
answers
1k
views
How to cause awk to fail if previous command (tar) also fails?
I have a bit of an issue with the following command which extracts a tar file & prints how many files have been extracted every second:
tar -xvf some_tar.tar -C a/directory | awk 'systime() > ...
- 273
0
votes
1
answer
59
views
Shell: Why is there a "duplicate process " on using a pipe and tee?
I'm not very familiar with bash process, and was surprised to see a "duplicate" process running while using pipe and tee. Can you help me understand this?
In the following, the path to the ...
- 1
0
votes
0
answers
39
views
How to redirect the output of a command running inside a loop on a remote machine (via ssh) to my local machine without breaking the loop?
I am running a piped ssh command on my Ubuntu 20.04 PC, where I am sshing into a remote machine, listening to a specific port and processing each incoming message as shown below:
ssh ${...
- 1
1
vote
0
answers
46
views
Why doesn't doas wget -qO- archive.tar.gz|doas tar -xz work as with sudo?
sudo wget -qO- https://go.dev/dl/go1.20.1.linux-amd64.tar.gz|sudo tar -xz works fine.
doas wget -qO- https://go.dev/dl/go1.20.1.linux-amd64.tar.gz|doas tar -xz does not work. I get no input, it just ...
- 561
1
vote
1
answer
87
views
checksum a pipeline
I want to construct a huge blob of data (a backup of a sort) and send it over the network (ssh or rsync) to another host. There is enough space on the remote for the data, but not on the local host, ...
- 518
5
votes
3
answers
303
views
Processing a continuous single line of data with stream processing in bash pipeline?
I am debugging an embedded server that outputs a continuous single line of text to a specified network port. There is no newline anywhere in the stream but it is text data and I would like to format ...
- 193
1
vote
2
answers
63
views
Tying to replace a file content with the machine current IP address
I have a yaml config file(cli.yaml). with structure
database:
host: "IP"
Now, I just want to replace the file content, Or more precisely THE IP string with my PC's current IPV4 address.
...
- 111
0
votes
1
answer
49
views
Do shells have a command equivalent of the "|" pipe symbol? I need something like that in Emacs org-mode tables
I like to construct commands which use the | pipe symbol in org tables and | symbol messes them up. Do bash et al have a text command that is the equivalent of the symbol?
Are there alternative ways ...
- 7,235
2
votes
6
answers
165
views
Only pipe output if at least n lines
I often want to grep output from processes that include a few header lines. But if grep would remove all actual content lines, I don't want to display the header lines. For this to work I would need a ...
- 254
0
votes
1
answer
116
views
Extract first n bytes from .tar.gz and output as a .tar.gz in a single command
I have a .tar.gz as input and want to extract the first 128 MiB of it and output as a .tar.gz in a single command. I tried:
sudo tar xzOf input.tar.gz | sudo dd of=output bs=1M count=128 iflag=...
- 3,476
0
votes
1
answer
66
views
Convert a file being downloaded by firefox (.part) into a named pipe?
Slightly unusual usecase: I have a download from firefox which is actually a stream of data. I want to process it in (near) real-time, without saving the data (it's high volume and would fill my drive ...
- 103
1
vote
1
answer
461
views
Usage of named pipe as data source for netcat
I want to use netcat as a TCP-server that reads data from a named pipe. For that I did the following:
Step 1. Created a pipe and the server that uses it as a source
mkfifo /tmp/all.pipe
nc -k -l 8080 &...
- 1,235