Chains the standard streams of a series of commands

learn more… | top users | synonyms

7
votes
4answers
113 views

How can I filter the contents of a tar file, producing another tar file in the pipe?

Consider a single tar file from an external system which contains some directories with various attributes which I want to retain such as permissions, mtimes, etc. How can I easily take a subset of ...
0
votes
1answer
22 views

Having extra parameters in piping or redirection

I have two programs and I want to pipe them into each other but they take extra parameters: prog1 <inputfile> <outputfile> min max prog2 <inputfile> <outputfile> min max ...
0
votes
1answer
15 views

cat SQL files and pass to import schema

I am trying to import a series of .SQL files which map one file per DB object into mysql without calling mysql 50 times, something like: mysql -u root < cat schema/db1/*.sql This isn't working ...
19
votes
9answers
3k views

Grep lines but let the first line through [duplicate]

Consider the following output from df. Filesystem Size Used Avail Use% Mounted on /dev/sda1 23G 6.1G 16G 29% / udev 10M 0 10M 0% /dev tmpfs 397M 420K ...
31
votes
4answers
2k views

Should I care about unnecessary cats?

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
25
votes
5answers
2k views

Why pipe to cat only to redirect? [duplicate]

I occasionally see things like: cat file | wc | cat > file2 Why do this? When will the results (or performance) differ (favourably) from simply: cat file | wc > file2
1
vote
3answers
69 views

pipe in shell with redirection

My understanding is that a pipe in shell like a | b redirects b's stdin to a's stdout. However, I tried ls | less < somefile, the behavior is very strange. less shows the output of ls concatenated ...
3
votes
1answer
127 views

How do I append an item to an array in a pipeline? [duplicate]

This script should simply add a value to an array through a loop and then show all items of an array. #!/bin/bash data_file="$1" down=() counter=0 cat $data_file | while read line; do \ ...
2
votes
3answers
36 views

Who forks the processes of piped commands in the shell?

Suppose this command in the shell : ls|less Now we know that ls and less are run as two different processes. We know that the shell forks and creates a child process which load the ls program using ...
6
votes
1answer
87 views

Extra output on input redirection

I have a folder with three files: $ ls a b c If I pipe the output of ls to wc, I get the right result: $ ls | wc -l 3 However, when I specify the input to wc as the output of ls, I get extra ...
0
votes
1answer
40 views

piping a password located in a file to the ssh command [duplicate]

So I need to login as a lot of different users to a server using ssh and I'm tired of copying the password from the files, which contain the passwords for the users, each time. I can't change it to ...
10
votes
2answers
2k views

Why does process substitution result in a file called /dev/fd/63 which is a pipe?

I am trying to understand named pipes in the context of this particular example. I type <(ls -l) in my terminal and get the output as, bash: /dev/fd/63: Permission denied. If I type cat <(ls ...
12
votes
3answers
750 views

No colored output in less for the ls command

If I run ls I get colored output which I find pretty handy for quickly getting a glance of the kind of file. When I try to pipe it to less even with the -r and -R flags the coloring always get lost. I ...
28
votes
6answers
10k views

How to make bidirectional pipe between two programs?

Everyone knows how to make unidirectional pipe between two programs (bind stdout of first one and stdin of second one): first | second. But how to make bidirectional pipe, i.e. cross-bind stdin and ...
4
votes
3answers
550 views

Behaviour of “|” pipe in linux with “>” output redirection

I am executing command ls > a.txt | sort > b.txt This command is doing the below things : executing ls sorting it creating a.txt and storing sorted output to a.txt creating b.txt , but its ...
0
votes
1answer
55 views

How to use pipe to assign a variable [duplicate]

I am new to bash script. Probably this is very easy but for me it's not working. How can I assign value of $x1+$x2 to bc by piping. here in my code $bc holds no value. echo -n 'value x1 : ' read x1 ...
-2
votes
2answers
51 views

Pipe Named Fifo [closed]

I am implementing a named pipe to write and read data between two processes in cpp. The first process acquires a feature point from an image in real time while the second reads this feature point. It ...
165
votes
11answers
61k views

Turn off buffering in pipe

I have a script which calls two commands: long_running_command | print_progress The long_running_command prints a progress but I'm unhappy with it. I'm using print_progress to make it more nice ...
1
vote
1answer
22 views

unzip a file and pipe to “ddrescue” command

I have a zipped image file, and need use "ddrescue" command instead "dd" to do something like this: unzip -p ubuntuXXXX.img.zip | sudo dd bs=2M of=/dev/sdb But "ddrescue" needs the image file as ...
4
votes
2answers
145 views

uniq and bash for loop not writing to stdout before stdin closing (for one-line website visitor notification system)

I'm trying to trigger a beep on the PC speaker for every unique visitor of a website. After some brainstorming, it seemed to be possible with one line: for e in `ssh me@mymachine "tail -n 1 -f ...
0
votes
4answers
71 views

piping commands not working as expected I need explanation

I am trying to do the below touch ~/scratches/scr6.txt | cat ~/scr.txt > ~/scratches/scr6.txt | cat /dev/null > ~/scr.txt I don't know why scr6.txt is always empty, however doing the below ...
16
votes
8answers
5k views

How to pipe output from one process to another but only execute if the first has output?

How can I rewrite this command to only email if there is output from the mailq | grep? mailq | egrep 'rejected|refused' -A 5 -B 5 | mail -s 'dd' email@email Is this even possible on one line?
9
votes
3answers
3k views

Piping STDERR vs. STDOUT

According to "Linux: The Complete Reference 6th Edition" (pg. 44), you can pipe only STDERR using the |& redirection symbols. I've written a pretty simple script to test this: #!/bin/bash echo ...
14
votes
2answers
332 views

Resource usage using pipe and here string

We can get the same result using the following two in bash, echo 'foo' | cat and cat <<< 'foo' My question is what are the difference between these two as far as the resources used are ...
3
votes
1answer
49 views

Round-tripping text through zip, unzip fails with 'extra bytes' error on OS X [duplicate]

I'm trying to zip streaming text, and unzip it later. A minimal example is: echo "Hey, could you zip this for me?" | zip hello.zip - ; unzip -p hello.zip This fails, with the following output: ...
15
votes
3answers
2k views

How do I “cat and follow” a file?

A file is being sequentially downloaded by wget. If I start unpacking it with cat myfile.tar.bz2 | tar -xj, it may unpack correctly or fail with "Unexpected EOF", depending on what is faster. How to ...
2
votes
0answers
31 views

Deadlock on communicating with multiple children [migrated]

This is a followup to the previous question: I have a parent.py: from sys import argv from random import randrange from subprocess import Popen, PIPE lines = int(argv[1]) procs = int(argv[2]) cmd = ...
1
vote
1answer
49 views

Deadlock on read/wait [closed]

My process deadlocks. master looks like this: p=Popen(cmd, stdin=PIPE, stdout=PIPE) for ....: # a few million p.stdin.write(...) p.stdin.close() out = p.stdout.read() p.stdout.close() exitcode = ...
1
vote
1answer
21 views

CD with redirect to logger does not work

On Linux, the following cd works fine so cds into tmp: / # cd /tmp tmp # But this does not work (stays at root, nothing appears in /var/log/messages). Why not? / # cd /tmp 2>&1 | ...
4
votes
2answers
590 views

Passing parsed output of sed to find (in this direction)

Well, I think you can find dozens of questions on this platform how to pipe find output to sed, but I haven't found anything for the reverse direction so far. What I want to do is modify my input, and ...
2
votes
0answers
26 views

Pipe a .BIN file into XModem transfer

I need to send a .bin file to a modem via XModem transfer for firmware upgrading. If the modem is connected to a local computer, the command would be the following: sz -X -k -b -vvv - ...
1
vote
2answers
102 views

How to while read line with a pipe

I am currently trying read a csv file with BASH. The 10th [$f10] value of the csv file must be greater then 0 for a given machine class and machine ID [m_class] and [m_id] (these are user defined) . ...
0
votes
1answer
50 views

Bash : rewriting a file which is read in same line

I'm working on a bash script for partially automating router connection in command line. I'm a newbie in Bash. This is part where I have a problem: #!/bin/bash STRING2="ESSID" MYPWD=$(pwd) sudo ...
5
votes
2answers
181 views

How to send all output to `logger` in POSIX shell?

I'd like to log standard output and standard error separately in .xprofile using logger. In Bash I think that would look something like this: exec 1> >(logger --priority user.notice --tag ...
5
votes
3answers
4k views

Running piped bash script in background

I'm attempting to build a monitoring script to watch localhost communication using netcat. I have two scripts that I've built, one to start the monitoring loop and one for the loop itself. They are as ...
3
votes
1answer
54 views

Joined pipelines

Considering a routine such as this one: alpha() { echo a b c |tr ' ' '\n'; } which outputs a stream, I would like to take the output stream, transform it, and paste it with the original output ...
1
vote
2answers
38 views

Piping contents of multiple files between programs while keeping contents separate

I would like to read in a number of files and pipe their output to subsequent programs while still maintaining them as individual pipelines of data. program1 *.txt | program2 | program3 folder I ...
0
votes
1answer
38 views

ssh over unix pipe or pty device [closed]

Ok, I don't even know how to put this into words. I have a /dev/myDevice which automatically connects to some ssh server when opened. How do I connect to it with ssh? Is this even sane? If it's insane ...
1
vote
4answers
189 views

Grep for a string in file without using pipe

I want to grep for a word in a file in the last n lines without using the pipe. grep <string> filename enables to search the filename for a string. But, I want to search for a string in the ...
2
votes
4answers
155 views

How to influence a bash while loop with a timed global variable?

I have a while loop in a bash script which should do something different at the beginning and at every 5 second interval. Any previous loop is allowed to complete. The 5s interval is indicated by the ...
3
votes
1answer
88 views

Possible to pipe command using su via ssh

I'm trying to write a python script to remotely update the login passwords for a set of account I administer - I can already successfully use python to pipe remote commands via the stdin of an ssh ...
0
votes
3answers
35 views

How to delete files from a directory based on the contents of a gzipped tar file?

Given that I: have a directory that contains over 1000 files have a gzip'ed tar file that contains a subset of those files (x.tgz) What is the single command line (if it is possible) that will ...
5
votes
3answers
216 views

How to forward between processes with named pipes?

The /tmp/in, /tmp/out and /tmp/err are named pipes, already created and opened by some process (for reading, writing and writing, respectively). I would like to create a new process that pipes its ...
0
votes
1answer
40 views

netstat pipe count host

Unix pipe problem: how to process multiline input to output the line and insert some piped processes. echo "111 22222222" | awk '{print $1 " " $2 " " piped_processes_using_2nd_argument}' A bit of ...
2
votes
0answers
450 views

avconv/ffmpeg output through /dev/ttyAMA0

I tried to pipe streaming video from video4linux2 webcam connected to a Raspberry Pi through /dev/ttyAMA0 to a computer using an Arduino as a intermediary. First of all, is it possible? Because I was ...
7
votes
2answers
10k views

Why is my variable local one 'while read' loop, but not in another seemingly similar loop? [duplicate]

Why do I get different values for $x from the snippets below? #!/bin/bash x=1 echo fred>junk ; while read var ; do x=55 ; done <junk echo x=$x # x=55 .. I'd expect this result x=1 cat ...
1
vote
4answers
81 views

optimize command with or and pipe to parse the output of ifconfig

I have this command line, but it's doubling the grep and awk ifconfig eth1 2> /dev/null | grep "inet " | awk '{gsub("addr:","",$2); print $2 }' || ifconfig eth0 2> /dev/null | grep "inet " | ...
53
votes
4answers
51k views

How big is the pipe buffer?

As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote: Another reason to avoid | true is that if the command ...
5
votes
1answer
252 views

Logging bash output to a file

I'm trying to build vlc which is rather complicated and dependent on having the right packages installed, and keeps choking on errors. To trace all my steps I wish to output what I'm doing to a ...
0
votes
2answers
69 views

viewing printf after pipe (or subshell?)

Ok, this is hard to distill to a single question because it really depends on how I'm going to troubleshoot it. The following script is meant to pipe mount.cifs commands to a while loop that enters ...