Hot answers tagged

292

awk '{$1=$1;print}' or shorter: awk '{$1=$1};1' Would trim leading and trailing space or tab characters1 and also squeeze sequences of tabs and spaces into a single space. That works because when you assign something to one of the fields, awk rebuilds the whole record (as printed by print) by joining all fields ($1, ..., $NF) with OFS (space by default). ...


279

There are 3 common ways of doing this: Pipefail The first way is to set the pipefail option (ksh, zsh or bash). This is the simplest and what it does is basically set the exit status $? to the exit code of the last program to exit non-zero (or zero if all exited successfully). $ false | true; echo $? 0 $ set -o pipefail $ false | true; echo $? 1 $...


182

You are confusing two very different types of inputs. Standard input (stdin) Command line arguments These are different, and are useful for different purposes. Some commands can take input in both ways, but they typically use them differently. Take for example the wc command: Passing input by stdin: ls | wc -l This will count the lines in the output of ...


165

In Bash, you can use the command1 <( command0 ) redirection syntax, which redirects command0's stdout and passes it to a command1 that takes a filename as a command-line argument. This is called process substitution. Some programs that take filename command-line arguments actually need a real random-access file, so this technique won't work for those. ...


158

In ./binary < file binary's stdin is the file open in read-only mode. Note that bash doesn't read the file at all, it just opens it for reading on the file descriptor 0 (stdin) of the process it executes binary in. In: ./binary << EOF test EOF Depending on the shell, binary's stdin will be either a deleted temporary file (AT&T ksh, zsh, bash......


155

watch 'command | othertool | yet-another-tool'


113

You can do it by telling wget to output its payload to stdout (with flag -O-) and supress its own output (with flag -q): wget -qO- your_link_here | tar xvz - To specify a target directory: wget -qO- your_link_here | tar xvz - -C /target/directory Update If you happen to have GNU tar wget -qO- your_link_here | tar --transform 's/^dbt2-0.37.50.3/dbt2/' -...


87

Yet another way to turn on line-buffering output mode for the long_running_command is to use the script command that runs your long_running_command in a pseudo terminal (pty). script -q /dev/null long_running_command | print_progress # (FreeBSD, Mac OS X) script -q -c "long_running_command" /dev/null | print_progress # (Linux)


82

You are almost there. In your last command, you can use -I to do the ls correctly -I replace-str Replace occurrences of replace-str in the initial-arguments with names read from standard input.  Also, unquoted blanks do not terminate input items; instead the separator is the newline character.  Implies -x and -L 1. So, with find . -type ...


80

If you have tee ./app | tee >(grep A > A.out) >(grep B > B.out) >(grep C > C.out) > /dev/null (from here) (about process substitution)


75

For grep, sed and awk you can force output to be line buffered. You can use: grep --line-buffered Force output to be line buffered.  By default, output is line buffered when standard output is a terminal and block buffered other-wise. sed -u Make output line buffered. See this page for more information: http://www.perkin.org.uk/posts/how-to-fix-stdio-...


72

The command can be condensed like so if you're using GNU sed: $ sed 's/^[ \t]*//;s/[ \t]*$//' < file Example Here's the above command in action. $ echo -e " \t blahblah \t " | sed 's/^[ \t]*//;s/[ \t]*$//' blahblah You can use hexdump to confirm that the sed command is stripping the desired characters correctly. $ echo -e " \t blahblah \t " |...


66

Use a named pipe. By way of illustration: mkfifo fifo echo -e "hello world\nnext line\nline 3" > fifo The -e tells echo to properly interpret the newline escape (\n). This will block, ie, your shell will hang until something reads the data from the pipe. Open another shell somewhere and in the same directory: cat fifo You'll read the echo, which ...


62

Using GNU sed: sed -i '1d;$d' Element_query How it works : -i option edit the file itself. You could also remove that option and redirect the output to a new file or another command if you want. 1d deletes the first line (1 to only act on the first line, d to delete it) $d deletes the last line ($ to only act on the last line, d to delete it) Going ...


61

This solution works without using bash specific features or temporary files. Bonus: in the end the exit status is actually an exit status and not some string in a file. Situation: someprog | filter you want the exit status from someprog and the output from filter. Here is my solution: ((((someprog; echo $? >&3) | filter >&4) 3>&1) | ...


60

This almost made me wince. You might want to stop pointing that shotgun at your foot. Basically any kind of parsing of ls is going to be more complicated and error-prone than established methods like find [...] -exec or globs. Unless someone installed a troll distro for you, your shell has Tab completion. Just type rm google and press Tab. If it doesn't ...


51

Try this: pgrep name | xargs kill If you use pgrep name | kill, the ouput of pgrep name is feed to stdin of kill. Because kill does not read arguments from stdin, so this will not work. Using xargs, it will build arguments for kill from stdin. Example: $ pgrep bash | xargs echo 5514 22298 23079


51

It's because you're doing it wrong. You're using bs=1M but reading from stdin, pipe, will have smaller reads. In fact, according to dd, you didn't get a single full read. And then you have conv=sync which complements incomplete reads with zeroes. 0+15281 records in 15280+0 records out dd received 0 full and 15281 incomplete reads, and wrote 15280 full ...


50

That's nothing to do with grep - it's because the pipe | redirects the standard output stream stdout whereas the Permission denied messages are in the standard error stream stderr. You could achieve the result you want by combining the streams using 2>&1 (redirect the stream whose file descriptor is 2 to the stream whose file descriptor is 1) so that ...


50

Since source (or .) takes a file as argument, you could try process substitution: source <(echo something)


48

The data doesn’t need to be stored in RAM. Pipes block their writers if the readers aren’t there or can’t keep up; under Linux (and most other implementations, I imagine) there’s some buffering but that’s not required. As mentioned by mtraceur and JdeBP (see the latter’s answer), early versions of Unix buffered pipes to disk, and this is how they helped ...


47

I tend to use this: command1 | xargs -I{} command2 {} Pass output of command1 through xargs using substitution (the braces) to command2. If command1 is find be sure to use -print0 and add -0 to xargs for null terminated strings and xargs will call command2 for each thing found. In your case (and taking the sed line from @Janos): command1 -p=aaa -v=bbb -i=...


47

Here's a simple example program that illustrates Stéphane Chazelas' answer using lseek(2) on its input: #include <stdio.h> #include <sys/types.h> #include <unistd.h> int main(void) { int c; off_t off; off = lseek(0, 10, SEEK_SET); if (off == -1) { perror("Error"); return -1; } c = getchar(); ...


46

When you do <(some_command), your shell executes the command inside the parentheses and replaces the whole thing with a file descriptor, that is connected to the command's stdout. So /dev/fd/63 is a pipe containing the output of your ls call. When you do <(ls -l) you get a Permission denied error, because the whole line is replaced with the pipe, ...


46

vipe is a program for editing pipelines: command1 | vipe | command2 You get an editor with the complete output of command1, and when you exit, the contents are passed on to command2 via the pipe. In this case, there's no command1. So, you could do: : | vipe | pandoc -o foo.pdf Or: vipe <&- | pandoc -o foo.pdf vipe picks up on the EDITOR and ...


45

Well, its fairly "easy" with named pipes (mkfifo). I put easy in quotes because unless the programs are designed for this, deadlock is likely. mkfifo fifo0 fifo1 ( prog1 > fifo0 < fifo1 ) & ( prog2 > fifo1 < fifo0 ) & ( exec 30<fifo0 31<fifo1 ) # write can't open until there is a reader # and ...


45

Found this clever answer in a similar question at stackoverflow (echo -e "cmd 1\ncmd 2" && cat) | ./shell_executable This does the trick. cat will pump in the output of echo into input stream of shell_executable and wait for more inputs until EOF.


45

There is a practical difference. curl -sSL https://get.docker.com/ | sh starts curl and sh at the same time, connecting the output of curl with the input of sh. curl will carry out with the download (roughly) as fast as sh can run the script. The server can detect the irregularities in the timing and inject malicious code not visible when simply downloading ...


45

When the data producer (tar) tries to write to the pipe too quickly for the consumer (lzip) to have time to read all of it, it will block until lzip has had time to read what tar is writing. There is a small buffer associated with the pipe, but its size is likely to be smaller than the size of most tar archives. There is no risk of filling up your system's ...


44

Named pipes (fifo) have four three advantages I can think of: you don't have to start the reading/writing processes at the same time you can have multiple readers/writers which do not need common ancestry as a file you can control ownership and permissions they are bi-directional, unnamed pipes may be unidirectional * *) Think of a standard shell | ...


Only top voted, non community-wiki answers of a minimum length are eligible