Local, timestamped logging of all ssh commands?

Question

How can I keep a local, timestamped record of all remote commands I use in ssh (command-line openssh client started through bash)?

Requirements:

• Essential:

  • 100% client-side without relying on server logging
  • Configured or installed per-user with logs stored in the user's home directory.
  • Support for distinguishing between multiple simultaneous sessions with various users and hosts.
  • Non-intrusive (no need to activate it each time and does not interfere significantly with using ssh)

• High priority:

  • Either output is not logged or filtered out as much as possible
  • Either password entries are not logged or the file is encrypted
  • Indicates the actual commands used (after tab/ history completion, backspaces, CTRL+C, etc... have been processed)

• Nice to have:

  • Also logs commands in chained sessions (commands entered during remote ssh or su <user> sessions)
  • Session start and end should be logged
  • A simple bash-based, non-root solution would be best (perhaps an alias or bash wrapper script for the ssh command?)

My skill level:

• I am not new to programming, but am still learning bash and the "Linux way", so code samples with brief explanations would be most appreciated.

Possible strategies

• keylogger -- Problem: logs passwords, does not log tab/ history completion (see glenn's answer)
• screen with scrollback dumping once per second and diff between them to find new scrollback lines -- Problem: how can this be implemented in an useful automated way?
• ssh "$@" | tee >(some_cleaner_function >> $logfile) -- Problem: cannot handle multiline commands or history in chained sessions, careful cleanup needed (see my answer)
• A combination of some of the above

An example

The following SSH session:

user@local:~$ ssh user@remote
Last login: Tue Jun 17 16:34:23 2014 from local
user@remote:~$ cd test
user@remote:~/test$ ls
a  b
user@remote:~/test$ exit

Might result in a log at ~/logs/ssh.log such as:

2014-06-17 16:34:50   [user@remote - start]
2014-06-17 16:34:51   [user@remote] cd test
2014-06-17 16:34:52   [user@remote] ls
2014-06-17 16:34:53   [user@remote] exit
2014-06-17 16:34:53   [user@remote - end]

Or, perhaps a separate log will be created for each session with the command line used to start the session at the top of the file.

Answer 1

I was intrigued by your question. I wasn't originally going to give an answer but I got hooked.

This uses expect and it's really a key logger.

#!/usr/bin/expect -f

proc log {msg} {
    puts $::fh "[timestamp -format {%Y-%m-%d %H:%M:%S}]: $msg"
}

set ssh_host [lindex $argv 0]
set ::fh [open "sshlog.$ssh_host" a]

log "{session starts}"

spawn ssh $ssh_host

interact {
    -re "(.)" {
        set char $interact_out(1,string)
        if {$char eq "\r"} {
            log $keystrokes
            set keystrokes ""
        } else {
            append keystrokes $char
        }
        send -- $char
    }
    eof
}

log "{session ends}"

Notes:

• it appends to a file with the ssh destination in the name
• it is a key logger: if you have not set up ssh keys, you get the user's password in the log file
• it is foiled by tab completion: if the user types uptTab (for the uptime command), you'll get "upt\t" in the log file, not "uptime"
• it grabs characters in "raw" mode: if the user is a bad typist, you'll get lots of ^? (backspace characters) in the log file.

Answer 2

I am currently using the bash script below. It has many problems, but it is the only solution I have found that addresses all of the requirements, priorities, and "nice to haves" (at least most of the time).

This answer discusses why logging ssh sessions locally is so difficult.

Issues with the script I found so far:

1. Multiline commands cause issues:

   • If you page through a multiline item in the remote history (with up/ down keys), it will log a history item instead of the latest command. You can avoid this by deleting from bash history any multiline commands immediately after they are used.
   • Only the first line of multiline commands are logged.

2. Chained sessions (using ssh or su commands on remote end) cause history scrolling to record scrolled-passed commands instead of the actual commands used

3. The regular expressions can be improved and may need to be modified for certain environments:

   • I cheat by converting nonprinting characters with cat -v before cleaning. As a result, valid content may be removed if you ever use strings like ^[[ in your commands.
   • Sometimes you get extra logged input before the command, such as if you page through the history very fast. This is generally followed by a "^M" before the actual command and thus could be stripped away if desired.
   • Other control characters sometimes occur. I am leaving them all in for now until I know which are safe to remove. ^M as I just mentioned is useful for detecting invalid logged input, and ^C would tell you if the command was aborted.
   • The prompt regex may need to be modified for particular prompts, and I could imagine that different remote environments may have different control character patterns.

4. No ssh command bash completion, such as for hostname. You can get bash completion if you alias this script to ssh with alias ssh="sshlog"

Script source and installation:

To install, paste the following into ~/bin/sshlog and make executable. Call with sshlog <ssh command options>. Optionally alias to 'ssh' in the user's .bashrc file.

#!/bin/bash
# A wrapper for the ssh command that produces a timestamped log of all ssh commands
declare -r logfile=~/logs/ssh.log
declare -r description="sshlog-${$} ${@}"
declare -r TAB=$'\t'

logdir=`dirname ${logfile}`
[ -d ${logdir} ] || mkdir "${logdir}";

clean_control_chars() {
    while IFS= read -r line; do
        # remove KNOWN control characters. Leave the rest for now.
        # line=$(echo "${line}" | sed 's/\^\[\[K//g')  # unkown control character: ^[[K
        # line=$(echo "${line}" | sed 's/\^\[\[[0-9]\+[P]//g')  # these are generated by up/down completion - e.g. ^[[2P
        line=$(echo "${line}" | sed 's/\^\[\[[0-9]*[A-Z]//g')  # all other ^[[..
        # replay character deletions (backspaces)
        while [[ $(echo "${line}" | grep -E --color=never '.\^H') != "" ]]; do
            line=$(echo "${line}" | sed 's/.\^H//')
        done
        # remove common control characters
        line=$(echo "${line}" | sed 's/\^M$//')  # remove end of line marker from end
        line=$(echo "${line}" | sed 's/^\^G//g')  # remove start marker from start
        # remove ^G from other locations - possibly a good idea
        # line=$(echo "${line}" | sed 's/\^G//g')
        # remove all other control characters - not recommended (many like ^C and ^M indicate which section was processed/ ignored)
        # line=$(echo "${line}" | sed 's/\^[A-Z]//g')
        echo ${line};
    done
}

filter_output() {
    while IFS= read -r line; do
        # convert nonprinting characters and filter out non-prompt (in Ubuntu 14.04 tests, ^G indicates prompt start)
        line=$(echo "${line}" | cat -v | grep -Eo '[\^][G].*[\$#].*')
        [[ ${line} != "" ]] && echo "${line}"
    done
}

format_line() {
    while IFS= read -r line; do
        raw=${line};
        line=$(echo "${line}" | clean_control_chars);
        prompt=$(echo "${line}" | grep -Po '^.*?(\$|#)[\s]*')
        command=${line:${#prompt}}
        timestamp=`date +"%Y-%m-%d %H:%M:%S %z"`
        echo -e "${timestamp}${TAB}${description}${TAB}${prompt}${TAB}${command}"
    done
}

echo "Logging ssh session: ${description}"
echo "[START]" | format_line >> ${logfile}
/usr/bin/ssh "$@" | tee >(filter_output | format_line >> ${logfile})
echo "[END]" | format_line >> ${logfile}

Example log content:

2014-06-29 23:04:06 -0700   sshlog-24176 remote [START]
2014-06-29 23:04:12 -0700   sshlog-24176 remote oleg@remote:~$  cd test
2014-06-29 23:04:13 -0700   sshlog-24176 remote oleg@remote:~/test$     ls
2014-06-29 23:04:14 -0700   sshlog-24176 remote oleg@remote:~/test$     exit
2014-06-29 23:04:14 -0700   sshlog-24176 remote [END]

Answer 3

How about strace -o /tmp/ssh_log -ff -s8192 -T -ttt -fp $(pidof sshd)? This logs all ssh sessions. You may need a tool to parse the log subsequently, or just use grep, awk etc.

• -f: trace forked children
• -ff: log each child separately to ssh_log.PID
• -s8192: increase string logging limit (if needed)
• -T -ttt: microsecond stamping in seconds since the Epoch
• -p N: attach to pid N

Answer 4

I have a less complicated answer, and surely not a keylogger. I don't get your point of being server log independent (this means that all the actions need to be taken to the server and all logs are server side logs), and thus I thought that a good idea is to pass to system wide bashrc a prompt command like:

PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -t "$USER[$$] $SSH_CONNECTION")'

In debian you should edit the file: /etc/bash.bashrc and in centos the file: /etc/bashrc

If you want to start logging for the session you are in, you have to source the file you have edited, for example execute:

source /etc/bash.bashrc

in a debian system or

source /etc/bashrc

in a centos system.

From now on, every command, of every ssh session will be logged at /var/log/syslog on a debian system, and at /var/log/messages on a centos system.

In case you want to log them on a separate file and not mess up with other log files you can use:

PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -p local6.info -t "$USER[$$] $SSH_CONNECTION")'

instead of the previous PROMPT_COMMAND example and then configure the rsyslogd as needed.

For example at a Debian System edit the /etc/rsyslog.conf file: change the line :

.;auth,authpriv.none           -/var/log/syslog

to

.;auth,authpriv.none,local6           -/var/log/syslog

and add the following line to the end of the file:

local6.info                     /var/log/history.log

then execute:

touch /var/log/history.log && /etc/init.d/rsyslog restart