Accurately overwriting previous lines of bash terminal output when text is wider than terminal (e.g. wraps mutliple lines)?

Question

TL;DR - I know how to overwrite lines of output normally but none of the methods I've used previously (e.g. printf '\e[1A\e[2K') or have found online seem to work when the line being overwritten is longer than width of terminal (e.g. a line-wrap was triggered). Disabling line-wrap effectively truncates the displayed text. Is there some other trick or tool or some way to handle this scenario that I'm missing?

I have a bash script that I share between my desktop (Fedora) and my phone (Termux on Android). Functionality-wise, I don't have any issues and everything works as expected. But script is fairly lengthly and the terminal output is a bloody mess. Recently, I learned that I could use ANSI Escape Codes from bash to overwrite a previous line of output and significantly clean up the output from my script, while still having an idea of its progress and seeing whatever error if it encounters one. My understanding of these is still pretty basic but from my testing it seems that printf recognizes \e as the start of an ASCII escape and then based on this, ESC[#A "moves cursor up # lines" and ESC[2K "erase the entire line".

Anyway, where I ran into an issue was that I was expecting all but the last line to be overwritten and instead multiple other lines were still being displayed. Initially, I thought this was due to some Termux bug but I later was able to confirm is due to the size (width) of the terminal (I am able to recreate the issue in gnome-terminal by resizing my window or by increasing the length of the text). Basically, what I am seeing is that if the line of output that I wish to overwrite is longer than the width of the terminal, then it appears that the line "wraps" the remaining text onto a new line and the overwrite only replaces the wrapped portion of the text.

Here is a snippet that recreates the issue I'm having in my script:

# create an array with variable-length texts to simulate status messages
arrStatusTexts=(  );
for i in {10..200..10}; do
    arrStatusTexts+=("$(printf '%*s\n' $i ' ' | tr ' ' X)");
done
 
# print out status at each step, overwriting output of each previous step as we go
echo "";
echo "--------------------";
echo "Steps of process"
echo "--------------------";
for ((i=0; i < ${#arrStatusTexts[@]}; i++)); do
    [[ 0 != "$i" ]] && printf '\e[1A\e[2K';
    printf 'Step %s of %s: %s\n' "$(($i + 1))" "${#arrStatusTexts[@]}" "${arrStatusTexts[$i]}";
done

From my desktop terminal window, before running the above, I get:

$ stty size
34 135

Edit: From both my desktop and from termux, my TERM variable shows as:

$ echo "$TERM"
xterm-256color

What I would like to see as the final output after running the above for loop is something like:

Step 20 of 20: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

What I actually see after for loop completes is:

Step 13 of 20: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Step 14 of 20: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Step 15 of 20: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Step 16 of 20: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Step 17 of 20: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Step 18 of 20: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Step 19 of 20: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Step 20 of 20: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Basically it is overwriting steps 1-12 correctly because those are smaller than the width of my terminal window. But for steps 13 onward, the length of each line "wraps" and only the wrapped portion is cleared.

---

Edit 2022-Sep-01: Found more oddities. Based on an answers here and here, using the \e[6n sequence, I was attempting to get the cursor position prior to each printf statement in order to see if that would somehow be useful.

altering the above to:

# from: https://unix.stackexchange.com/a/183121/379297
function pos () {
    local CURPOS
    read -sdR -p $'\E[6n' CURPOS
    CURPOS=${CURPOS#*[} # Strip decoration characters <ESC>[
    echo "${CURPOS}"    # Return position in "row;col" format
}
export -f pos;
arrCursorPos=(  );
for ((i=0; i < ${#arrStatusTexts[@]}; i++)); do
    # get cursor position
    arrCursorPos+=("$(pos)");
    printf 'Step %s of %s: %s\n' "$(($i + 1))" "${#arrStatusTexts[@]}" "${arrStatusTexts[$i]}";
done
for ((i=0; i < ${#arrCursorPos[@]}; i++)); do
    echo "cursor pos $(($i + 1)): '${arrCursorPos[$i]}'";
done

The output I got for the status messages is the same as above but here's the output of the 2nd array which holds cursor positions:

cursor pos 1: '19;1'
cursor pos 2: '20;1'
cursor pos 3: '21;1'
cursor pos 4: '22;1'
cursor pos 5: '23;1'
cursor pos 6: '24;1'
cursor pos 7: '25;1'
cursor pos 8: '26;1'
cursor pos 9: '27;1'
cursor pos 10: '28;1'
cursor pos 11: '29;1'
cursor pos 12: '30;1'
cursor pos 13: '31;1'
cursor pos 14: '33;1'
cursor pos 15: '34;1'
cursor pos 16: '34;1'
cursor pos 17: '34;1'
cursor pos 18: '34;1'
cursor pos 19: '34;1'
cursor pos 20: '34;1'

At first, I thought this was not working correctly because I was getting the same pos for indexes 15-20. After clearing screen (Ctrl+L) and rerunning a few times I saw different output

cursor pos 1: '11;1'
cursor pos 2: '12;1'
cursor pos 3: '13;1'
cursor pos 4: '14;1'
cursor pos 5: '15;1'
cursor pos 6: '16;1'
cursor pos 7: '17;1'
cursor pos 8: '18;1'
cursor pos 9: '19;1'
cursor pos 10: '20;1'
cursor pos 11: '21;1'
cursor pos 12: '22;1'
cursor pos 13: '23;1'
cursor pos 14: '25;1'
cursor pos 15: '27;1'
cursor pos 16: '29;1'
cursor pos 17: '31;1'
cursor pos 18: '33;1'
cursor pos 19: '34;1'
cursor pos 20: '34;1'

And realized what is going on. For the last few array elements, it is getting to the last row (in my case 34 - as is reported in col1 by stty size). At that point, any new lines that are output cause the displayed text to scroll and I still end up on the last line (34). So this method does seem like it might be a reliable way of keep track of initial cursor position.

I also tried an alternate approach (suggested here, here, and here involving exec < /dev/tty and stty. Using the function here and modifying the snippet to:

function extract_current_cursor_position () {
    export $1
    exec < /dev/tty
    oldstty=$(stty -g)
    stty raw -echo min 0
    echo -en "\033[6n" > /dev/tty
    IFS=';' read -r -d R -a pos
    stty $oldstty
    eval "$1[0]=$((${pos[0]:2} - 2))"
    eval "$1[1]=$((${pos[1]} - 1))"
}
export -f extract_current_cursor_position;
arrCursorPos=(  );
for ((i=0; i < ${#arrStatusTexts[@]}; i++)); do
    # get cursor position
    extract_current_cursor_position pos1
    arrCursorPos+=("${pos1[0]} ${pos1[1]}");
    printf 'Step %s of %s: %s\n' "$(($i + 1))" "${#arrStatusTexts[@]}" "${arrStatusTexts[$i]}";
done
for ((i=0; i < ${#arrCursorPos[@]}; i++)); do
    echo "cursor pos $(($i + 1)): '${arrCursorPos[$i]}'";
done

Rerunning this after clearing screen, I got:

cursor pos 1: '10 0'
cursor pos 2: '11 0'
cursor pos 3: '12 0'
cursor pos 4: '13 0'
cursor pos 5: '14 0'
cursor pos 6: '15 0'
cursor pos 7: '16 0'
cursor pos 8: '17 0'
cursor pos 9: '18 0'
cursor pos 10: '19 0'
cursor pos 11: '20 0'
cursor pos 12: '21 0'
cursor pos 13: '22 0'
cursor pos 14: '24 0'
cursor pos 15: '26 0'
cursor pos 16: '28 0'
cursor pos 17: '30 0'
cursor pos 18: '32 0'
cursor pos 19: '32 0'
cursor pos 20: '32 0'

This one seems like it would work too. Though I am not sure why the extract_current_cursor_position function subtracts 2 from the y and 1 from the x values. I would probably need to look into that or remove it the subtractions

I still have to look into ncurses options (e.g. tput). I did already check that the ncurses package is at least available on Termux but I'll come back and fill in more info as I test more.

---

The obvious changes that I could make are:

1. Don't change my script and just live with the multi-line clutter like I've been doing. But I would strongly prefer to fix my output, unless it becomes too much work.

2. Shorten all the status texts and reduce printing variables in output till everything is smaller than the smallest screen's width (stty size on my phone reports 17 48 so e.g. limit everything to 48 chars per line). While I don't mind some refactoring and putting in more effort, the idea of doing hundreds of text changes seems very tedious and teaches me nothing about what is actually going on. I would rather save this as a last resort for if there turns out not to be a better way. That and this would probably lose meaningful info or require me making other compromises to how things are displayed.

3. Same as 2 but use print -v msg to put the output in a variable then print truncated text e.g. printf '%s\n' "${msg:0:48}". Same issues as with 2 but I would also likely lose some meaningful info.

4. Similar to above but rather than truncating, just keep track of the previous message's length and divide by terminal width to determine the number of printf '\e[1A\e[2K'; statements I should use. Still a bit of work but less than if I had to edit every message and should give a better end result.

Curious if there is an easier way to go about this that I am missing. Maybe some way way to "grep" text that was already printed and clear to a specific offset from current position (adding a unicode char or asterisk or something to beginning of each status line would be pretty easy search/replace). Or some command or builtin that I am unaware of? My google fu hasn't turned up any obvious solutions besides what I listed above.

I don't need a fully POSIX-compliant solution; just one that works in bash with standard tools (python/perl/awk one-liners are all fair game... but rewriting my entire bash script into one of those is not). Considering this domain usually deals with desktop questions, I'm not expecting familiarity with Termux but if the solution requires a graphical session (e.g. won't work on ssh) or a tool only available on x86_64 architectures (Termux uses ARM builds) then it probably won't work. Most common bash/linux tools seem to work fine there tho. My desktop currently has bash 5.1.8 (and I will be upgrading to Fedora 36 soon) and Termux has the ARM version of bash 5.1.16.

Answer 1

About half-way through writing a solution to your question I realised that this is why (n)curses terminal virtualisation is so much easier: it hides all the nasty terminal dependant details (well most of them, well some of them). Using terminfo directly is painful - but better than raw escape sequences.

This is bash code. It shouldn't be hard to rewrite as sh though.

# Output a printf style format string and arguments and return the cursor
# to the beginning of the line. DO NOT use newline `\n`.
#
lineOut() {
    local rows cols len lines

    # Number of rows/columns on the terminal device
    rows=$(tput lines)
    cols=$(tput cols)

    # Output
    printf "$@"

    # How many lines we wrote
    len=$(printf "$@" | unexpand -a | wc -m)
    lines=$(( len / cols ))

    if tput am
    then
        # Cursor does not wrap when writing to the last column
        len=$(( len - (cols * lines) ))
        [[ $len -eq 0 ]] && (( lines-- ))
    fi

    # Move up the necessary number of lines to column 1
    printf '\r'
    for (( ; lines > 0; lines-- )); do tput cuu1; done
}


for ((i=0; i < ${#arrStatusTexts[@]}; i++)); do
    lineOut 'Step %s of %s: %s' "$(($i + 1))" "${#arrStatusTexts[@]}" "${arrStatusTexts[$i]}";
    sleep 1
done
printf '.\n'

The terminfo codes used by tput are documented in man 5 terminfo. Running with a terminal type that does not have the necessary sets of control codes (try export TERM=dumb, for example) the solution degrades cleanly.

---

While working on a solution I found u7 (user7), which happens to trigger the "What is your cursor position?" question of the terminal:

# Magic to read the current cursor position (origin 1,1)
tput u7; read -t1 -srd'[' _; IFS=';' read -t1 -srd'R' y x

It's no longer relevant to the solution proposed here but it may come in useful as a bonus.

Answer 2

One method is to record where the cursor was, then jump back to that point as need be. This could be done manually with escape sequence wrangling, but that's more work. This method may not play well with scrolling text, though Curses does have a scrollok call to influence what happens when something prints off the end of the window. Probably depends on what you want to see in the scrollback verses simply having text in a non-scrolled window; maybe play around with the alternate screen to show one thing there and something else afterwards?

#!/usr/bin/env perl
use strict;
use warnings;
# or you could do it the hard way with XTerm Control Sequences: ask the
# terminal where the cursor is, manually parse the result, etc
use Curses;
initscr;

my ( $y, $x );    # where we started printing from to jump back to

move int rand(4), 0;    # put the cursor somewhere (start point)
getyx $y, $x;           # record this

my $i = 0;
while ( ++$i < 10 ) {
    clrtobot;       # maybe subsequent messages are shorter? clear all below
    addstring $i x ( $COLS + int rand 8 );    # noise that wraps a line
    refresh;
    sleep 1;
    move $y, $x;                              # back to where we started from
}

endwin;