Tag Info

New answers tagged

1

The terms you need in order to use some google-fu are 'redirection' which is what you're doing in process < input.txt and 'piping' which is what you're doing in cat input.txt | process Once you know this, you can find articles like this one http://askubuntu.com/questions/172982/what-is-the-difference-between-redirection-and-pipe which answers your ...


3

The redirection to file.txt at the end of your paste command is truncating your file before paste has a chance to read it. Try echo 2 | paste file.txt - > file2.txt or if you have sponge installed echo 2 | paste file.txt - | sponge file.txt


0

Here is a post summarizing Unix output streams: http://www.devcodenote.com/2015/04/unix-output-streams.html A snippet from the post: There are 3 standard output streams: STDIN - Standard Input - Writes from an input device to the program STDOUT - Standard Output - Writes program output to screen unless specified otherwise. STDERR - Standard Error Output - ...


4

Yes, it slows things down. And it basically does have a queue of unwritten data, though that's actually maintained by the kernel—all programs have that, unless they explicitly request otherwise. For example, here is a trivial pipe using pv, which is nice because it displays transfer rate: $ pv -s 50g -S -pteba /dev/zero | cat > /dev/null 50GiB ...


1

In a command where we have lot of pipes redirecting their outputs further, is there any way to get to know the value of echo $? till half of it's execution or till certain number of pipes, as opposed to the overall status of the whole command's? In bash, there's a PIPESTATUS variable, which is an array containing the exit status of each command in the ...


0

I think I understand what you are trying to accomplish: For each hit to the web site, which is logged by the web server: If the visit is "unique" (how do you define this??) log the entry and send an audible notification. The trick is how you define "unique". Is it by URL, by IP address, by cookie? Your approach with awk was arguably the right way to go, ...


1

You have this command ls -lrt --time-style=+"%b %d %Y %H:%M:%S" /some/path/*.dat|head -1|tr -s " "|cut -d " " -f 9|date --date - +%s The problem with this is that date does not expect its parameters via stdin, so you need to split it into two parts. You've already attempted this in the second part of your Question: var1=$(ls -lrt --time-style=+"%b %d %Y ...


0

All of your problems are explained in Why does my shell script choke on whitespace or other special characters?, but I'll repeat the relevant parts here. First, $var in a shell script does not mean “the value of the variable var”, it means “take the value of the variable var, split it into words and interpret each word as a wildcard pattern”. To avoid these ...


0

You may be inadvertently shell-expanding the contents of your archive. Try quoting your variable. Replace echo $ARCHIVE_CONTENTS | tar xzv -C $TMPDIR with echo "$ARCHIVE_CONTENTS" | tar xzv -C $TMPDIR You also may need to avoid appending a newline with echo -n


1

why not use a here document like in the old days ? #!/bin/bash export TMPDIR=`mktemp -d /tmp/selfextract.XXXXXX` ## other stuff base64 -d <<EOF | tar xvf -C $TMPDIR ## base64 encoded EOF you produce base64 encoded this way tar cf - my_dir | base64 > /some/place (including it int self extract archive is left to the reader)


0

A number of issues that might benefit from review. sudo dd if=/dev/sda bs=64k | pv --size 1.5t | dd of=/dev/sdb Firstly, you can (vastly) increase the block size and correspondingly increase the throughput. I often use bs=32M. The order of the parameters to dd does not matter, so: sudo dd if=/dev/sda bs=1M Next, it doesn't matter whether you specify ...


1

That's a job for awk. #!/usr/bin/awk -f $0 == "@@inline-code-start" {pipe = 1; next} $0 == "@@inline-code-end" {pipe = 0; close("nl"); next} pipe {print | "nl"} !pipe {print} When the script sees the start marker, it notes that it should start piping into nl. When the pipe variable is true (nonzero), the output is piped into the nl command; when the ...


0

Parse the logfile line for line and suppress all \n's. When you see a new entry, first write \n except for the first time. You said Each entry has some stuff at the start (datetime, duration) but you did not give an example. Ok, I will call it NEW_ENTRY, you can modify. inStatement=0 cat logfile | while read -r line; do if [[ ${inStatement} = 0 ]]; then ...


0

A shell script that uses sed to output chunks of non-demarcated lines and feed demarcated chunks of lines into a filter program: #!/bin/bash usage(){ echo " usage: $0 <input file>" } # Check input file if [ ! -f "$1" ]; then usage exit 1 fi # Program to use for filtering # e.g. FILTER='tr X -' FILTER='./filter.sh' # Generate arrays ...


-1

Thanks for all the great ideas. I've come up with my own solution by keeping track of the subsection in a temp file and piping it all at once to my external command. This is very similar to what Supr suggested (but with a shell variable instead of temp file). Also, I really like the idea of using sed, but the syntax for this case seems a bit over the top for ...


0

OK, first off; I understand that you’re not looking for a way to number the lines in sections of your file.  Since you haven’t given an actual example of what your filter might be (other than nl), let’s suppose that it is tr "[[:lower:]]" "[[:upper:]]" i.e., convert text to all upper case; so, for an input of line A line B @@inline-code-start line X line ...


2

If your goal is to send the entire code block to a single process instance then you could accumulate the lines and delay piping until you reach the end of the code block: #!/bin/bash acc="" while read line do if [[ $line == @@inline-code-start* ]] then active=true acc="" elif [[ $line == @@inline-code-end* ]] then active=false # ...


4

One possibility is to do this with the vim text editor. It can pipe arbitrary sections through shell commands. One way to do this is by line numbers, using :4,6!nl. This ex command will run nl on the lines 4-6 inclusive, achieving what you want on your example input. Another, more interactive way is to select the appropriate lines using line-selection mode ...


7

I would agree with you - it probably is a generic problem. Some common utilities have some facilities for handling it, though. nl nl, for example, separates input into logical pages as -delimited by a two character section delimiter. Three occurrences on a line all alone indicate the start of a heading, two the body and one the footer. It replaces any ...


2

Edit added an option to define a user-provided filter #!/usr/bin/perl -s use IPC::Open2; our $p; $p = "nl" unless $p; ## default filter $/ = "\@\@inline-code-end\n"; while(<>) { chomp; s/\@\@inline-code-start\n(.*)/pipeit($1,$p)/se; print; } sub pipeit{my($text,$pipe)=@_; open2(my $R, my $W,$pipe) || die("can open2"); local $/ = ...


2

The simplest fix I can think of is to not use nl but count the lines yourself: #!/usr/bin/env bash while read line do if [[ $line == @@inline-code-start* ]] then active=true elif [[ $line == @@inline-code-end* ]] then active=false elif [[ $active = true ]] then ## Count the line number let num++; ...


2

From write(2), Errors       ⋮ EPIPE fd is connected to a pipe or socket whose reading end is closed.  When this happens the writing process will also receive a SIGPIPE signal.  … In other words, pipes and sockets, while not the same thing, are similar enough that Unix uses the same mechanism to inform a writing process that there is ...


0

SSH sets up a "pipe" between the two servers for the communication channel. It broke, so the SSH session collapsed.


0

Using gawk you could use a (subset of) PCRE expression as register separator (RS), define a different output register separator (ORS) and replace \n. Example: gawk 'BEGIN {RS="[ ]*;\n"; ORS="\n===\n"} {gsub("\n","\\n"); print} ' in this example: registers are separated by [ ]*;\n in the input registers are separated by "\n===\n" in the ...


0

"|"` AKA, "Pipes"....pipes let you use the output of a program as the input of another one" . So if let's say we do this: user$ cat SHAHashing.java | grep main public static void main(String[] args)throws Exception as you can see the cat executed first displaying the page's contents "THE OUTPUT" , and then grep for the string main . Now lets assume ...


3

The processes are started at the same time, and will run concurrently, but they don't have to stop at the same time. The shell will consider the entire pipeline to have terminated (and display a new prompt) when both processes have terminated. If command2 ends before command1 does (or closes its standard input stream), and command1 then attempts to write ...


2

You can use -I to define a place-holder which will be replaced with each value of the arguments fed to xargs. For example, ls -1 | xargs -I '{}' echo '{}' will call echo once per line from ls's output. You'll often see '{}' used, presumably because it's the same as find's place-holder. In your case, you also need to pre-process file's output to extract ...


3

You know, I'm not convinced you necessarily need a repetitive feedback loop as your diagrams portray, so much as maybe you could use a persistent pipeline between coprocesses. Then again, it may be there isn't too much of a difference - once you open a line on a coprocess you can implement typical style loops just writing information to and reading ...


3

In general I would use a Makefile (command make) and try to map your diagram to makefile rules. f1 f2 : f0 command < f0 > f1 2>f2 To have repetitive/cyclic commands, we need to define a iteration policy. With: SHELL=/bin/bash a.out : accumulator cat accumulator <(date) > a.out cp a.out accumulator accumulator: touch ...


9

You can use a FIFO for this, created with mkfifo. Note however that its very easy to accidentally create a deadlock. Let me explain that—take your hypothetical "circular" example. You feed a command's output to its input. There are at least two ways this might deadlock: The command has an output buffer. It's partially filled, but hasn't been flushed ...


15

Circular I/O Loop Implemented with tail -f This implements a circular I/O loop: $ echo 1 >file $ tail -f file | while read n; do echo $((n+1)); sleep 1; done | tee -a file 2 3 4 5 6 7 [..snip...] This implements the circular input/output loop using the sine algorithm that you mentioned: $ echo 1 >file $ tail -f file | while read n; do echo ...


4

There exists several variants of a watch command, some that spawn a shell to interpret a command line made of the concatenation of the arguments passed to watch (with space characters in between). In those you can do: watch 'ls | shuf' same as: watch ls '|' shuf (those watch actually run: "/bin/sh", ["sh", "-c", "ls | shuf"] and are quite dangerous in ...


2

You don't say what shell are you using. From the behaviour you are describing it's likely zsh. If you have a look in its man page you would notice how redirections are handled. Note that a pipe is an implicit redirection; thus cat bar | sort <foo is equivalent to cat bar foo | sort (note the order of the inputs). Otherwise, regular ...


2

Your understanding is not quite correct. In a | b the stdout output of process a connected through a pipe to stdin of process b. The problem with your code is that with an additional redirection from somefile to process b you will use two different methods at the same time to connect to stdin of process b. Don't do that! The question is; what do you try to ...


1

I found solution: #!/bin/bash mkfifo /etc/z.conf ( while (true) do ssh x@y cat /etc/z.conf > /etc/z.conf done ) &


1

wget -qO - |gzip -c > file_name.gz -c for stdout . that's '>' used. the file get from wget serialize to file_name.gz using standard output library. -qO to send output file


3

You will need to store the output in a variable to accomplish this. Here is an example: if output=$(cmda); then printf '%s' "$output" | cmdb fi


3

In the first line, xargs still waits for the second argument or an end of the input. After pressing Ctrl-D xargs continues with the rest and you will see the 5th x as single argument. This example may explain the behavior: (echo "x x x x x"; sleep 5; echo "x") | xargs -n2 Output: x x x x x x # after 5 seconds After the 6th x in the second echo ...


13

Do you really want to tar the file or are you looking for downloading a file into a compressed form. Tarring a file is just bundling (uncompressed) files into an (uncompressed) archive. If you want to download a file into a compressed file you can use: wget -qO - <url>|gzip -c - > file.gz


4

This is not going to work the way you want it to. A file (obviously with a filename) needs to be stored in the tar. That bit (the filename) is obviously missing if you just pipe the contents of the download to tar. I don't see any way to tell tar that it should pack stdin and specify a filename for that. That said, I really do not say a way to achieve that ...


8

You can't deal with input streams in that way. It is designed to deal with files. If you had managed to create an archive as you describe, what would it look like? How would you untar it? There would be no filename to create, just data. I think your best bet is to get the file and tar it in two separate commands. If you don't want the file to remain on ...


2

bash has both an interactive mode and a batch mode. It enters the corresponding mode depending if stdin is a terminal or not. bash # interactive cat|bash # non-interactive: stdin is a pipe not a terminal cat|bash -i # explicitly request interactive mode In interactive mode bash will print a prompt (configurable by the PS1 variable) and also set ...


6

You can try things out with a script such as #!/bin/sh for fd in 0 1 2; do if [ -t $fd ]; then echo $fd is a TTY; fi done Running this I see that: if the script is run on its own, all three FDs are TTYs if the script is run at the start of a pipeline, stdin and stderr are TTYs if the script is run in the middle of a pipeline, stderr is a TTY if the ...


4

LC_ALL=C </dev/urandom \ tr '\0-\377' '[0*128][1*]' | dd ibs=50 cbs=10 conv=unblock count=1 That will convert all input ascii bytes (which will be all bytes because LC_ALL=C is specified) into one of either 0 or 1 on an even distribution. The first 128 bytes between \0 and \177 are converted to zeroes and the \200-\377 to ones - and so you get to use ...


1

then I tried to fill a file with this stream (and end the process of filling by ctrl+c) cat /dev/urandom | tr -dc 01 > foo when I count the numbers of lines of the so created foo file I get 0 lines. cat foo | wc -l 0 Both cat and tr buffer their output. When you press Ctrl+C, any data that's still in either command's buffer is lost. You ...


7

How about fold? It's part of coreutils... $ tr -dc 01 < /dev/urandom | fold -w 30 | head -n 5 001010000111110001100101101101 000101110011011100100101111000 111010101011100101010110111001 111011000000000101111110110100 110011010111001110011010100011 Or if that's not available, some flavour of awk: $ tr -dc 01 < /dev/urandom | awk \$0=RT RS=.\{,30} | ...


1

To add a newline during the generation process do: { process-without-terminating-newline ; echo ;} > outfile To add it to an existing file do: echo >> outfile



Top 50 recent answers are included