Why does history not overwrite but append when histappend is set to off in Bash?

Question

Several days ago, I had one Terminal tab doing ssh, and the other doing a lot of work locally. And then I restarted the machine (on a Mac with Mavericks).

When I ran Terminal again, I found that all the command history of the 2nd tab was not there. There was only the ssh command.

So then I searched for how to "merge" the history somehow, and found that we need to do a shopt -s histappend in our ~/.bashrc file so that history will append, instead of "overwrite" -- we don't want one set of history overwriting the other set; we want to append.

However, when I go to the OS X Yosemite box, and used shopt, I saw that histappend is off, but when I opened up two tabs, one doing echo abc, and the other doing echo def, and exited them, I re-opened Terminal again and issued a history command, and saw both the echo abc and echo def commands.

And then I ran Ubuntu 2014-10 on VirtualBox and did similar things, and still saw both history recorded (I first ran shopt -u histappend in both Bash to set the option to off first).

So what mechanism is doing that really? Then histappend has no significance, if setting it to on or off doesn't matter?

I also commented out the shopt -s histappend in my .bashrc and restarted Terminal and tried again, and saw the history of both tabs able to combine... so this is really strange behavior and what can be causing it?

Answer 1

@David Dai is correct about this bash-help thread.

Here is the relevant bash code in C.

if (history_lines_this_session <= where_history () || force_append_history)
  {
    result = append_history (history_lines_this_session, hf);
    history_lines_in_file += history_lines_this_session;
  }
else
  {
    result = write_history (hf);
    history_lines_in_file = history_lines_this_session;
  }

shopt -s histappend sets force_append_history to non-zero (true).

history_lines_this_session is the number of commands you typed this interactive bash session. When you first start bash, it is 0. It increases by 1 each time you Enter a command that is saved to history.

where_history() is an index into the in-memory history list, starting at index 0. It points to the next index to write history to. For example, when you have 0 history, it will be index 0. When you have 6 lines of history, it will be index 6.

When you first start bash and you have history in your HISTFILE, where_history() increase but will be limited to <= HISTSIZE. So where_history() has a maximum value of HISTSIZE.

After reading the HISTFILE, where_history() will increase but history_lines_this_session will still be 0. When you Enter HISTSIZE + 1 number of commands, history_lines_this_session == HISTSIZE + 1 and where_history() == HISTSIZE (because it is capped at the maximum). At this point, (history_lines_this_session <= where_history()) == false so history will be overwritten if histappend is off.

You can test this out in your bash yourself. If histappend is off, once you run HISTSIZE + 1 number of commands, history_lines_this_session will exceed where_history() by 1 and HISTFILE will be overwritten. You have to make sure other commands aren't messing with your history for the test to work though (happens on macOS). Use the following commands to turn off commands that may mess with history:

trap - EXIT
PROMPT_COMMAND=
SHELL_SESSION_HISTORY=0 # for macOS only

Answer 2

I believe histappend is the behaviour set for one terminal.

Try the following in a new terminal:

shopt -s histappend
export HISTSIZE=1
export HISTFILESIZE=500

Then close the terminal. You will then see how your history file has the last 499 last commands it already had plus "export HISTFILESIZE=50", which you just typed. Your history was appended and then truncated to 500.

Now try this in another new terminal:

shopt -u histappend
export HISTIZE=1
export HISTFILESIZE=500

And again, close the terminal. You will see how now your terminal has just one command "export HISTFILESIZE=500".

The trick is that without histappend, the history file is overwritten with what there currently is in the terminal history (one command).

You won't notice it's behaviour if HISTFILE = HISTFILESIZE but I believe it has nothing to do with working on multiple terminals, this must be managed by something else...