3

I use ksh (version: sh (AT&T Research) 2020.0.0) on Ubuntu 20.04, and I use vi command line editing mode (set -o vi). This has worked fine for years, but recently I have noticed some oddities, and finally this morning it stopped working altogether.

  • I can see commands are no longer being saved in ~/.sh_histfile as of this morning
  • In the past, all commands I entered could be found by scrolling up with Esck or eg. searching with Esc/.
  • A while ago, this seemed to change to only remembering successful commands (which is a nuisance).

I haven't made any changes to my configuration yesterday (I think), but command line history works for the root user.

Permissions on ~/.sh_histfile are 600, and HISTFILE is not set, either for my normal user or root.

Any idea what has gone wrong - and of course how to fix it? Ideally I want to get back to the situation when anything I wrote on the command line and finished with Enter would be saved to ~/.sh_histfile.

Edit

The following variables are set:

$ set
_=export
COMP_CWORD=0
COMP_KEY=0
COMP_POINT=0
COMP_TYPE=0
COMP_WORDBREAKS=$'"\'@><=;|&(:'
DBUS_SESSION_BUS_ADDRESS='unix:path=/run/user/1000/bus'
DISPLAY=localhost:11.0
ENV=/home/jan.andersen/.kshrc
FCEDIT=/usr/bin/ex
HISTCMD=1
HOME=/home/jan.andersen
IFS=$' \t\n'
JOBMAX=0
KSH_VERSION=.sh.version
LANG=en_US.UTF-8
LESS=X
LINENO=1
LOGNAME=jan.andersen
MAILCHECK=600
MOTD_SHOWN=pam
OLDPWD=/home/jan.andersen
OPTIND=1
PATH=/home/jan.andersen/.local/bin:/usr/local/glassfish5/bin:/usr/local/glassfish5/glassfish/bin:/usr/local/texlive/2019/bin/x86_64-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PPID=3887
PS1=$'\'$ \''
PS2='> '
PS3='#? '
PS4='+ '
PWD=/home/jan.andersen
RANDOM=18082
SECONDS=17131.589
SHELL=/usr/bin/ksh
SHLVL=1
SH_OPTIONS=astbin=/opt/ast/bin
SSH_TTY=/dev/pts/1
TERM=xterm-256color
TMOUT=0
USER=jan.andersen
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_CLASS=user
XDG_SESSION_ID=10
XDG_SESSION_TYPE=tty

The local profile contains only default stuff and a coule of lines at the end, that I've added:

$ cat .profile
# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.

# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022

# if running bash
if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
        . "$HOME/.bashrc"
    fi
fi

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

#eval $(ssh-agent -s)
#ssh-add ~/.ssh/id_rsa
export SSH_AUTH_SOCK=${XDG_RUNTIME_DIR}/ssh-agent.socket
16
  • What's the HISTSIZE variable set to?
    – Kusalananda
    May 26 at 9:05
  • Any clues in environment? Typically do export or env from shell.
    – ibuprofen
    May 26 at 10:22
  • The output of env is not really helpful as variables affecting the shell's behaviour do not need to have been exported as environment variables. Consider updating your question with your ~/.profile file and possibly any other file that you may be sourcing at shell startup (e.g. the file that the $ENV value mentions, if any).
    – Kusalananda
    May 26 at 11:06
  • You have ENV=/home/jan.andersen/.kshrc. Could you please show the contents of /home/jan.andersen/.kshrc?
    – Kusalananda
    May 26 at 14:41
  • @Kusalananda It doesn't exist
    – j4nd3r53n
    May 26 at 15:03
1

This is not verified by other sources, but merely a look at the case.

NB! This relates to ksh-2020.

There is a bug in the handling of history when history gets above a certain size (bytes). It manifests itself in that history length is 1 (aka zero) and stays there. If one reach the limit that triggers the bug while in ksh it manifest itself on next run.

Using something like:

PS1='$_pwd [!]\$ '

shows path + number of history entries.

In a normal run sh_histinit is called on start. For some reason, not tracked down, this function is called for every command when this bug is active. Further hist_write is normally called, to write the record, but this is not called when bug is active.

As a result one end up with a new file descriptor to .sh_hisory for every command. This is likely what you also saw in /proc/PID/fd as noted in comments. Further commands are not written to file.

Solution, at least here, is to archive the old history and start a new. Likely advisable to restart shell.

Without HISTSIZE defined, the limit for triggering the bug is small. A few hundred records. You can set it to a big number, like 50000, to "delay" it. In my tests it was then triggered around 32k lines with a size of 50000. Increased size to 500000 and then it worked as expected again.

As noted it is also likely advisable to install ksh93 instead of the 2020 version. From the looks of things the 2020 has stalled. (93 does not mean it is from 1993 - but based on that version. Likely 93u)

The 93u version does not have this bug.

If /etc/skel/.kshrc is not sourced you likely also want to copy that file to your home directory. It likely already have a HISTSIZE set, change that to a big value if you continue with the 2020 release.



Original "comment"

This is too long for a comment, so I write it as an "answer". (Assuming Linux)

You could possibly find a clue to what is going on by:


In one terminal run ksh normally. Get the pid of that shell.

Check /proc/PID/fd/. It should typically have open fd 3 to /home/username/.sh_history

In second shell do strace -p PID. In first shell when you enter k you should typically see something like this in the shell running strace.

Read history by k:

select(1, [0], NULL, NULL, NULL)        = 1 (in [0])
recvfrom(0, 0x7ffdcff6c0d0, 80, MSG_PEEK, NULL, NULL) = -1 ENOTSOCK (Socket operation on non-socket)
read(0, "k", 80)                        = 1
lseek(3, 0, SEEK_SET)                   = 0
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\201\1[ --help\n\0exit\n\0env\n\0\0ls\n\0q\n\0\0"..., 65536) = 258
lseek(3, 0, SEEK_END)                   = 258
write(2, "man foo\10\10\10\10\10\10\10", 14) = 14

Here:

read(0, "k", 80) = 1 read from STDIN, read 1 byte, "k" (The key entered)
read(3, "\201\1[ ... read 65536 bytes from fd 3, read history file. write(2, "man foo... write man foo, the previous command is shown.

Write history:

In the first shell enter a command. In log below I entered cd irc from ~/tmp

Excerpt from entering c<Enter>

write(2, "c", 1)                        = 1
select(1, [0], NULL, NULL, NULL)        = 1 (in [0])
recvfrom(0, 0x7ffdcff6c0d0, 80, MSG_PEEK, NULL, NULL) = -1 ENOTSOCK (Socket operation on non-socket)
read(0, "\r", 80)                       = 1
ioctl(2, TCGETS, {B38400 opost -isig -icanon -echo ...}) = 0
ioctl(2, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0
write(2, "\n", 1)                       = 1
lseek(3, 0, SEEK_END)                   = 270
lseek(3, 0, SEEK_CUR)                   = 270
lseek(3, 270, SEEK_SET)                 = 270
read(3, "", 65536)                      = 0
lseek(3, 0, SEEK_END)                   = 270
lseek(3, 0, SEEK_END)                   = 270
write(3, "cd irc\n\0", 8)               = 8
lseek(3, 0, SEEK_CUR)                   = 278
chdir("irc")                            = 0
lseek(3, 0, SEEK_END)                   = 278
lseek(3, 278, SEEK_SET)                 = 278
lseek(3, 278, SEEK_SET)                 = 278
read(3, "", 65536)                      = 0
write(2, "~/tmp/irc [36]$ ", 16)        = 16

Here the:
write(3, "cd irc\n\0", 8) = 8

is the line that writes to the history file.

7
  • We must assume that the user is not lying, so stracing the process should not show writing to or reading from ~/.sh_history. Your answer lacks a way of diagnosing anything or saying anything other than "that's not what happens".
    – Kusalananda
    May 26 at 11:10
  • @Kusalananda: Absolutely. But curious as to if it could yield some hint on what is happening. Does it for example for some unknown reason write to another file somewhere else ... - "luckily" the strace of a shell in this manner is not very verbose (not much noise).
    – ibuprofen
    May 26 at 11:11
  • @Kusalananda I find the answer very useful. The list of open files in /proc/PID/fd is strange - there are 7 descriptors pointing to my history file, starting from 11 to 17, and in the trace I see the shell access fd=3 (and succeeds!), but apparently it isn't attached to a file (?)
    – j4nd3r53n
    May 26 at 11:14
  • @j4nd3r53n: That sounds strange. At the same time I have SHELL=/bin/bash so not the best test environment. If no-one enters an answer I can have a look at a VM later ...
    – ibuprofen
    May 26 at 12:06
  • @ibuprofen Yes, it is odd - the only difference from root's environment is that it has only one file descriptor poiting to .sh_history
    – j4nd3r53n
    May 26 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.