KSH history weirdness

Question

I'm running KSH on RHEL 2.6 on x86. KSH version is:

sh (AT&T Research) 93u+ 2012-08-01

Switching to Bash or back to Tcsh (which I happily used at my old job) is not possible.

The first annoying thing I noticed about KSH is that sometimes my history from different terminals (I have many terminals open at a time) got mixed together when scrolling through history with the up key. I then learned that there's a single HISTFILE shared by all my terminals. Fine, but still weird that they weren't always getting intermixed. I also noticed the HISTFILE wasn't always updated immediately. Eventually I decided that KSH was probably buffering some command history and periodically flushing the contents to HISTFILE. I couldn't find that behavior documented anywhere but it seems like a reasonable explanation.

I really want different history for each session, so I added this to my .kshrc file:

HISTFILE=~/.hist.$(tty | sed 's;.*/;;')

which is supposed to give a unique HISTFILE per terminal. It does that, which is great. But...these HISTFILES seem to periodically get cleared. Most of mine have a file size of 0 bytes right now.

Update: The individual HISTFILEs for all my open terminals are being cleared every 10 minutes. Like clockwork.

Anybody seen this before???

Update: Two years later and I am still no closer to an answer. It's been particularly annoying lately. Started poking around the ksh source but I'm a hardware designer and there aren't a lot of comments so this is pretty tough.

Well maybe this is a clue. Every line in my history files starts with a ^@ character (as rendered by emacs). I think that means 0x40 (@ character) + 0x80 (for the control bit) = 0xc0 = octal 300. The only special characters mentioned in history.c, as far as I can tell, are 0201 and 0202. My history files do start with \201 (again, as rendered by emacs, I guess this is octal).

Here's a sample of a typical history file, does this seem normal:

�ls
 pwd
  clear
  ls
 ls
 tty

Hmm those control characters don't display correctly. Transcribed as regular characters it looks like:

\201^Als
^@pwd
^@^@clear
^@^@ls
^@ls
^@tty
^@^@

Sometimes I get two of those ^@ characters and sometimes one, and I can't figure out why. This might not be a clue but I have no other leads right now.

UPDATE: SOLVED! Well almost. No sooner did I post that then I started searching for some other things in history.c. I saw this:

/*
 * clean out history file OK if not modified in HIST_RECENT seconds
 */
static int hist_clean(int fd)
{
    struct stat statb;
    return(fstat(fd,&statb)>=0 && (time((time_t*)0)-statb.st_mtime) >= HIST_RECENT);
}

and no surprise here, HIST_RECENT is set to 600, exactly the 10 minute interval I was looking for. I haven't quite figured out how to disable this yet, but it sure seems like intentional behavior, no matter how much I hate it.

Answer 1

Yes, ksh93 writes its command line history to file as you go, while bash writes it when the shell exits. Digging though some source code, it seems that the in-memory history buffer is 4k (HIST_BSIZE in the code), but I'm not able to figure out whether it flushes this when it's full or after a certain number of commands (whan may HIST_MARKSZ be doing in this code?).

And yes, many interactive sessions share history if they use the same HISTFILE setting. This is documented (my emphasis):

Command Re-entry.

The text of the last HISTSIZE (default 512) commands entered from a terminal device is saved in a history file. The file $HOME/.sh_history is used if the HISTFILE variable is not set or if the file it names is not writable. A shell can access the commands of all interactive shells which use the same named HISTFILE.

The history file will be kept to the size specified in HISTSIZE but it will not be truncated periodically as you experience. What's happening for you is, I think, that since you use tty to set a unique HISTFILE for each interactive shell session, exiting and restarting interactive sessions may or may not pick up the "correct" history file depending on what TTY the shell is connected to.

If a new TTY (never used with a session before) is allocated for a shell session, it's history file will naturally be empty.