Filter script output during runtime

Question

I have a script that runs about 10 other scripts in the background that constantly generate new output. I would like to keep that initial script running and be able to simply type in it to filter all the output displayed on the terminal in real time.

For the sake of simplicity let say each of these for loops represents one of my background scripts (The outputs contain color codes btw.):

for i in {0..1000} ; do
    echo -e "\033[0;31mHello world\033[0m $i"
    sleep 1
done &


for i in {0..1000} ; do
    echo -e "\033[0;34mHello world\033[0m $i"
    sleep 1
done &

My first idea was to redirect all the output of the background scripts into a file and then use the read command to get a search query and pass it to grep like so:

for i in {0..1000} ; do
    echo -e "\033[0;31mHello world\033[0m $i"
    sleep 1
done >> ./output.txt 2>&1 &

for j in {0..1000} ; do
    echo -e "\033[0;34mHello world\033[0m $j"
    sleep 1
done >> ./output.txt 2>&1 &

while true ; do
    read -p "Search: " query
    clear
    cat ./output.txt | grep "$query"
done

But there are several issues with that.

Update problem: The output won't update when the output.txt file changes. So you have to repeatedly search again to get up-to-date results.

Color code problem: If I search for 3 it will print all lines because the color codes contains a 3. I could of course completely filter out the colors like so: cat ./output.txt | sed 's/\x1b\[[0-9;]*m//g' | grep "$query", but I don't want to lose the color in the final output so it's not that easy.

Disappearing input problem: Even if I'd manage to get new output to be printed in real time after hitting enter for a search, the user won't be able to type (or see, to be more accurate) his/her next search properly. I would like the filter text the user is typing to always be visible.

Any ideas how this could be accomplished?

Answer 1

less -R

You can use the viewer less to

• see the ANSI colours with the option -R
• search forward for pattern with /pattern

• search backward for pattern with ?pattern

• Navigate with the arrow keys and PgUp, PgDn, Home, End.

---

• Please notice that your buffer is updated, when you press the End key :-)
• The colour code problem is solved.
• The input is not disappearing.

Use the following script for writing,

#!/bin/bash

for i in {0..1000} ; do
    echo -e "\033[0;31mHello world\033[0m $i"
    sleep 1
done >> output.txt 2>&1 &

for j in {0..1000} ; do
    echo -e "\033[0;34mHello world\033[0m $j"
    sleep 1
done >> output.txt 2>&1 &

Use this command line for reading/checking,

less -R output.txt

Shellscript with fifo, tail and grep

An alternative to the simple less command is the following small shellscript. It assumes that you can scroll enough in a terminal window to see also old output. It is possible to increase the buffer, if it is not big enough. I have the following lines in my `.bashrc', and I think they do what you want for this purpose,

HISTFILESIZE=1000000
HISTSIZE=10000

Shellscript to filter and check,

#!/bin/bash

# this function is called when Ctrl-C is sent ##########################

inversvid="\0033[7m"
resetvid="\0033[0m"

function trap_ctrlc ()
{
    # perform cleanup here

    echo " Press <Enter> to continue with another query or"
    echo -en "$inversvid"
    read -s -n1 -t5 -p " <x> to exit " ans
    echo -en "$resetvid"
    if [ "$ans" == "x" ]
    then
     # exit shell script
     # if omitted, shell script will continue execution
     echo ""
     exit
    else
     echo ""
    fi
}

##### main #############################################################

# initialise trap to call trap_ctrlc function
# when signal 2 (SIGINT) is received

trap "trap_ctrlc" 2

echo "Scroll the terminal window (maybe inscrease the buffer to store enough lines)

Interrupt viewing the filtered output with <ctrl c>"

if [ "$1" == "" ]
then
   read -p "Search: " query
else
   query="$@"
fi

# create fifo

mkfifo fifo

# run a loop for queries and filtered output

while true ; do
 if [ "$query" == "" ]
 then
    read -p "New search: " query
 fi

# mkfifo fifo
 clear
 tail -n 10000 -f output.txt > fifo &
 grep --color -E "$query" fifo
 sleep 0.1
 query=""
done

Please notice that this shellscript will see also the ANSI escape sequences, so for example if you want to filter for 0, you will see all lines (because there is a zero in the reset sequence). You can use extended regular expressions (the option -E of grep which makes it possible to specify what you want to search for, for example \ .?0 or0$.

(This is not a problem with less -R).