There is this log file that data coming in is continuous. What I wanted to happen is to tail -f this log file then redirect it to a file with the ff conditions

example of the log files content


in tailing, I want the redirection to start when it found the first pattern then stop the redirection as well as tailing when I found the second pattern. e.g

I want the redirection starts when it found pattern "ddddddddddddddd" then will stops when it found "hhhhhhhhhhhhhhh" the contents of the file created from redirection should be


In coding I'm thinking of something like this. But I don't have idea how to stop the redirection when it found the second pattern.

tail -f logfile > log.tmp
while grep "ddddddddddddddd" log.tmp
cat log.tmp > logfile

This is exactly what retail does. retail is tail with regular expressions, a tool I wrote for exactly the use case you have here. In your case, you'd use:

retail -f -r ddddddddddddddd -u hhhhhhhhhhhhhhh logfile > log.tmp

-f is the standard tail -f option. -r takes a regular expression to use to start the range of lines to include, and -u takes a regular expression for it to continue until. It begins at the last occurrence of the -r pattern and exits after printing the first occurrence of the -u pattern thereafter.

If you want to start from the first instance of the pattern you can use -b in combination with -r. Both regular expressions are EREs, without any implicit anchoring, but you can use ^ and $ to anchor the match as usual.

You can obtain and build retail with:

git clone https://github.com/mwh/retail.git
cd retail
make install

It installs into ~/.local/bin by default, but you can change that, or just copy the executable where you want.

retail is fully POSIX-compatible, though I don't recommend actually using it as your system tail.

  • is this a redhat package or debian? how can i install this?
    – nolram16
    Apr 24 '15 at 8:37
  • There are installation instructions between the horizontal rules. As far as I know there are neither Red Hat nor Debian packages. Apr 24 '15 at 8:39
  • can you send me the link and instructions how to install this? TIA
    – nolram16
    Apr 24 '15 at 8:42
  • Look about ten centimetres above your comment, where the grey box is. Apr 24 '15 at 8:45
  • 1
    @mikeserv: No -r at the moment, but maybe sometime. The POSIX compatibility was more of a why-not than anything. It has exactly the options mentioned above plus -c. Apr 24 '15 at 9:07

If I understand your demands correct this does what you expect:

awk '/ddddddddddddddd/,/hhhhhhhhhhhhhhh/ { print > "excerpt" } 1' infile | tail -f

awk works as filter on file "infile"; it prints everything in between the given patterns into a file "excerpt", and it also prints every line to standard output, which is then processed by tail -f as usual.

In case you want to use it with streamed data:

some_process | awk '/ddddddddddddddd/,/hhhhhhhhhhhhhhh/ { print > "excerpt" } 1'  | tail -f

(Note: Since tail -f is an interactive program it should come, if possible, always last in a pipeline.)

  • If the OP is considering tail -f at all then it suggests the source file is growing over time. In this situation awk would fall off the end of the partially created file and your tail -f would never see any more input.
    – roaima
    Apr 24 '15 at 9:50
  • @roaima; That's why I provided the second variant; don't create the file but pipe data directly from the generating process. If you want the logfile with all data anyway persistent then use tee: some_process | tee logfile | awk '...' | tail -f.
    – Janis
    Apr 24 '15 at 12:13
  • In neither example does the tail -f add any value
    – roaima
    Apr 24 '15 at 12:38
  • @roaima; Huh? - If you have a long continuously written logfile you want to follow it as it's written. Wasn't that the point of the request?! To explain the (optional) parts: the tee logfile is if you want a persistent complete logfile, the awk part is to filter the pattern-matched subsection and store it separately on the fly, and the tail -f is to watch the log interactively while it's generate.
    – Janis
    Apr 24 '15 at 12:49
  • Yes exactly! Contrast for F in seq 9; do echo "$F"; sleep 1; done | awk '/4/,/7/' | tail -f with and without the tail. Similarly, to simulate from a growing file, for F in seq 9; do echo "$F"; sleep 1; done >tmp & sleep 1 ; awk '/4/,/7/' tmp | tail -f also with and without the tail
    – roaima
    Apr 24 '15 at 14:01

Maybe you can use awk's range patterns here:

tail -f logfile | awk '/ddddddddddddddd/,/hhhhhhhhhhhhhhh/'

If you need to circumvent the SIGPIPE problem you can use socat instead of tail for the job:

socat -u file:logfile,ignoreeof "system:'stdbuf -o0 awk /ddddddddddddddd/,/hhhhhhhhhhhhhhh/'" > logfile.new
  • You might need an exit in the awk code to generate SIGPIPE back to the tail so that it also stopped
    – roaima
    Apr 24 '15 at 9:38
  • In that case I would use socat for the job.
    – FloHimself
    Apr 24 '15 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.