How can I filter certain commands from multi-line Bash history on logout?

Question

I've configured Bash to save multiline history entries with embedded newlines.

shopt -s cmdhist lithist
export HISTTIMEFORMAT='%F %T '

When I exit the shell, my history is appended to ~/.bash_history with timestamps separating the entries, like:

#1501293767
foo() {
echo foo
}
#1501293785
ls

I experimented with setting HISTIGNORE to filter out some commands,

HISTIGNORE='ls[ ]*:man[ ]*:cat[ ]*'

but quickly realized I prefer to include everything in the current session's (in memory) history, I just want to filter what gets written to disk for future sessions.

I'd like to leverage ~/.bash_logout to do this filtering (I've been looking at these answers to a couple related questions).

The multiline entries make filtering a little complicated. I need to identify entries where the first line following the timestamp matches one of the ignorable patterns, and exclude the entire entry, including the timestamp.

I've come up with a way to do this with awk, but my awk skills are not great, so I wonder if there's a better way?

filter_history() {
  tmpfile=$(mktemp)
  history -w $tmpfile

  awk '/^#[[:digit:]]{10}$/  { timestamp=$0; ignore=0; next }
       length(timestamp) > 0 && /^(ls|man|cat)([^[:alnum:]]|$)/ { timestamp=""; ignore=1 ; next }
       length(timestamp) > 0 { print timestamp; timestamp=""; print; next }
       ignore == 0           { print }' \
      $tmpfile >> $HISTFILE && rm $tmpfile
}

Edit: Actually, I can't think of a case where I'd want to filter out a multi-line entry. Even if it starts with ls, for instance, if it span multiple lines it's probably doing something interesting, and worth remembering.

Answer 1

The requirement to keep multi-line entries even if the first line matches an "ignore" pattern added some complexity. I ended up writing a finite-state machine in Awk to filter HISTFILE after it was written to disk (I couldn't find a way to trigger filtering before it had already been written).

~/.bashrc:

# if shell is interactive, filter history upon exit
if [[ $- == *i* ]]; then
  trap '$HOME/.bash_history_filter >/dev/null 2>&1 &' EXIT
fi

~/.bash_history_filter:

tmpfile=$(mktemp)
trap 'rm -f "$tmpfile"' EXIT

filter_script="$HOME/.bash_history_filter.awk"
persisted_history="${HISTFILE:-$HOME/.bash_history}"

if [[ -r "$filter_script" && -r "$persisted_history" ]]; then
  awk -f "$filter_script" "$persisted_history" > "$tmpfile"

  mv "$tmpfile" "$persisted_history"
fi

~/.bash_history_filter.awk:

/^#[[:digit:]]{10}$/ {
  timestamp = $0
  histentry = ""
  next
}
$1 ~ /^(ls?|man|cat)$/ {
  if (! timestamp) {
    print
  } else {
    histentry = $0
  }
  next
}
timestamp {
  print timestamp
  timestamp = ""
}
histentry {
  print histentry
  histentry = ""
}
{ print }

A couple related posts (here and here) lead me to suspect this could also be done using sed. I haven't figured that out yet, but I'd be curious to see how it compares.

Answer 2

for the completness , here is how i deal with bash history! (filter & backup)

however you filter or not the HISTFILE isn't enouth for someone who cares about his history, also having a big HISTFILE isn't recommanded , so sure someday you will need to keep your history somewhere else!

function shellHist () #find old hist
{ 
    if [[ -f ${HISTFILE}.${1} ]]; then
        cat ${HISTFILE}.${1};
    else
        grep -h "${@:-.}" ${HISTFILE}.*;
    fi
}
function shellHistBackup () 
{ 
    [[ -n ${HISTFILE} ]] || return;
    # pidKill0 && return; # do nothing if the job is actually running elsewhere
    {
    while read histLine; do
        if ! grep -q "^${histLine}$" ${HISTFILE}.${histLine%% *} 2> /dev/null; then
            echo "${histLine}" >> ${HISTFILE}.${histLine%% *};
        fi;
    done < ${HISTFILE}
    for fltr in ls cat any ; do
        rm ${HISTFILE}.${fltr}
    done
    } & # pidLog # related to pidKill0
}

i include shellHistBackup in "trap shellExit EXIT"

i hope this will be useful for somebody!