10

How can I keep a local, timestamped record of all remote commands I use in ssh (command-line openssh client started through bash)?

Requirements:

  • Essential:

    • 100% client-side without relying on server logging
    • Configured or installed per-user with logs stored in the user's home directory.
    • Support for distinguishing between multiple simultaneous sessions with various users and hosts.
    • Non-intrusive (no need to activate it each time and does not interfere significantly with using ssh)
  • High priority:

    • Either output is not logged or filtered out as much as possible
    • Either password entries are not logged or the file is encrypted
    • Indicates the actual commands used (after tab/ history completion, backspaces, CTRL+C, etc... have been processed)
  • Nice to have:

    • Also logs commands in chained sessions (commands entered during remote ssh or su <user> sessions)
    • Session start and end should be logged
    • A simple bash-based, non-root solution would be best (perhaps an alias or bash wrapper script for the ssh command?)

My skill level:

  • I am not new to programming, but am still learning bash and the "Linux way", so code samples with brief explanations would be most appreciated.

Possible strategies

  • keylogger -- Problem: logs passwords, does not log tab/ history completion (see glenn's answer)
  • screen with scrollback dumping once per second and diff between them to find new scrollback lines -- Problem: how can this be implemented in an useful automated way?
  • ssh "$@" | tee >(some_cleaner_function >> $logfile) -- Problem: cannot handle multiline commands or history in chained sessions, careful cleanup needed (see my answer)
  • A combination of some of the above

An example

The following SSH session:

user@local:~$ ssh user@remote
Last login: Tue Jun 17 16:34:23 2014 from local
user@remote:~$ cd test
user@remote:~/test$ ls
a  b
user@remote:~/test$ exit

Might result in a log at ~/logs/ssh.log such as:

2014-06-17 16:34:50   [user@remote - start]
2014-06-17 16:34:51   [user@remote] cd test
2014-06-17 16:34:52   [user@remote] ls
2014-06-17 16:34:53   [user@remote] exit
2014-06-17 16:34:53   [user@remote - end]

Or, perhaps a separate log will be created for each session with the command line used to start the session at the top of the file.

migrated from askubuntu.com Jun 30 '14 at 3:24

This question came from our site for Ubuntu users and developers.

  • It should also handle editors like nano or vim – daisy Sep 26 '17 at 2:48
4

I was intrigued by your question. I wasn't originally going to give an answer but I got hooked.

This uses expect and it's really a key logger.

#!/usr/bin/expect -f

proc log {msg} {
    puts $::fh "[timestamp -format {%Y-%m-%d %H:%M:%S}]: $msg"
}

set ssh_host [lindex $argv 0]
set ::fh [open "sshlog.$ssh_host" a]

log "{session starts}"

spawn ssh $ssh_host

interact {
    -re "(.)" {
        set char $interact_out(1,string)
        if {$char eq "\r"} {
            log $keystrokes
            set keystrokes ""
        } else {
            append keystrokes $char
        }
        send -- $char
    }
    eof
}

log "{session ends}"

Notes:

  • it appends to a file with the ssh destination in the name
  • it is a key logger: if you have not set up ssh keys, you get the user's password in the log file
  • it is foiled by tab completion: if the user types uptTab (for the uptime command), you'll get "upt\t" in the log file, not "uptime"
  • it grabs characters in "raw" mode: if the user is a bad typist, you'll get lots of ^? (backspace characters) in the log file.
  • Thank you very much for your answer. It is interesting that there doesn't seem to be an easy answer for this, perhaps native to the ssh client. Thank you for explaining the limitations; I think tab completion/ backspace character logging/ password logging are enough to prevent me from using this often. It also made me think more about my priorities, and I will clarify those in the question. – Oleg Jun 20 '14 at 17:48
  • The output from the spawned process can be parsed to extract the command you want. You'd need to know the user's prompt to make it easier. – glenn jackman Jun 20 '14 at 17:51
  • I use tab completion a lot. Doesn't this mean that those commands would not be logged? – Oleg Jun 20 '14 at 17:58
  • Ah, I see what you're saying. How would the output be parsed? The prompt could be entered somewhere as a configuration option. – Oleg Jun 20 '14 at 17:59
  • I added the list of possible solutions at the bottom of the question. ssh ... | tee -ai <logfile> works well to log input and output safely, but I don't know how to add timestamps and/ or filter out output in the background. – Oleg Jun 20 '14 at 18:11
2

I am currently using the bash script below. It has many problems, but it is the only solution I have found that addresses all of the requirements, priorities, and "nice to haves" (at least most of the time).

This answer discusses why logging ssh sessions locally is so difficult.

Issues with the script I found so far:

  1. Multiline commands cause issues:

    • If you page through a multiline item in the remote history (with up/ down keys), it will log a history item instead of the latest command. You can avoid this by deleting from bash history any multiline commands immediately after they are used.
    • Only the first line of multiline commands are logged.
  2. Chained sessions (using ssh or su commands on remote end) cause history scrolling to record scrolled-passed commands instead of the actual commands used

  3. The regular expressions can be improved and may need to be modified for certain environments:

    • I cheat by converting nonprinting characters with cat -v before cleaning. As a result, valid content may be removed if you ever use strings like ^[[ in your commands.
    • Sometimes you get extra logged input before the command, such as if you page through the history very fast. This is generally followed by a "^M" before the actual command and thus could be stripped away if desired.
    • Other control characters sometimes occur. I am leaving them all in for now until I know which are safe to remove. ^M as I just mentioned is useful for detecting invalid logged input, and ^C would tell you if the command was aborted.
    • The prompt regex may need to be modified for particular prompts, and I could imagine that different remote environments may have different control character patterns.
  4. No ssh command bash completion, such as for hostname. You can get bash completion if you alias this script to ssh with alias ssh="sshlog"

Script source and installation:

To install, paste the following into ~/bin/sshlog and make executable. Call with sshlog <ssh command options>. Optionally alias to 'ssh' in the user's .bashrc file.

#!/bin/bash
# A wrapper for the ssh command that produces a timestamped log of all ssh commands
declare -r logfile=~/logs/ssh.log
declare -r description="sshlog-${$} ${@}"
declare -r TAB=$'\t'

logdir=`dirname ${logfile}`
[ -d ${logdir} ] || mkdir "${logdir}";

clean_control_chars() {
    while IFS= read -r line; do
        # remove KNOWN control characters. Leave the rest for now.
        # line=$(echo "${line}" | sed 's/\^\[\[K//g')  # unkown control character: ^[[K
        # line=$(echo "${line}" | sed 's/\^\[\[[0-9]\+[P]//g')  # these are generated by up/down completion - e.g. ^[[2P
        line=$(echo "${line}" | sed 's/\^\[\[[0-9]*[A-Z]//g')  # all other ^[[..
        # replay character deletions (backspaces)
        while [[ $(echo "${line}" | grep -E --color=never '.\^H') != "" ]]; do
            line=$(echo "${line}" | sed 's/.\^H//')
        done
        # remove common control characters
        line=$(echo "${line}" | sed 's/\^M$//')  # remove end of line marker from end
        line=$(echo "${line}" | sed 's/^\^G//g')  # remove start marker from start
        # remove ^G from other locations - possibly a good idea
        # line=$(echo "${line}" | sed 's/\^G//g')
        # remove all other control characters - not recommended (many like ^C and ^M indicate which section was processed/ ignored)
        # line=$(echo "${line}" | sed 's/\^[A-Z]//g')
        echo ${line};
    done
}

filter_output() {
    while IFS= read -r line; do
        # convert nonprinting characters and filter out non-prompt (in Ubuntu 14.04 tests, ^G indicates prompt start)
        line=$(echo "${line}" | cat -v | grep -Eo '[\^][G].*[\$#].*')
        [[ ${line} != "" ]] && echo "${line}"
    done
}

format_line() {
    while IFS= read -r line; do
        raw=${line};
        line=$(echo "${line}" | clean_control_chars);
        prompt=$(echo "${line}" | grep -Po '^.*?(\$|#)[\s]*')
        command=${line:${#prompt}}
        timestamp=`date +"%Y-%m-%d %H:%M:%S %z"`
        echo -e "${timestamp}${TAB}${description}${TAB}${prompt}${TAB}${command}"
    done
}

echo "Logging ssh session: ${description}"
echo "[START]" | format_line >> ${logfile}
/usr/bin/ssh "$@" | tee >(filter_output | format_line >> ${logfile})
echo "[END]" | format_line >> ${logfile}

Example log content:

2014-06-29 23:04:06 -0700   sshlog-24176 remote [START]
2014-06-29 23:04:12 -0700   sshlog-24176 remote oleg@remote:~$  cd test
2014-06-29 23:04:13 -0700   sshlog-24176 remote oleg@remote:~/test$     ls
2014-06-29 23:04:14 -0700   sshlog-24176 remote oleg@remote:~/test$     exit
2014-06-29 23:04:14 -0700   sshlog-24176 remote [END]
0

I have a less complicated answer, and surely not a keylogger. I don't get your point of being server log independent (this means that all the actions need to be taken to the server and all logs are server side logs), and thus I thought that a good idea is to pass to system wide bashrc a prompt command like:


PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -t "$USER[$$] $SSH_CONNECTION")'

In debian you should edit the file: /etc/bash.bashrc and in centos the file: /etc/bashrc

If you want to start logging for the session you are in, you have to source the file you have edited, for example execute:


source /etc/bash.bashrc

in a debian system or


source /etc/bashrc
in a centos system.

From now on, every command, of every ssh session will be logged at /var/log/syslog on a debian system, and at /var/log/messages on a centos system.

In case you want to log them on a separate file and not mess up with other log files you can use:


PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -p local6.info -t "$USER[$$] $SSH_CONNECTION")'
instead of the previous PROMPT_COMMAND example and then configure the rsyslogd as needed.

For example at a Debian System edit the /etc/rsyslog.conf file: change the line :


.;auth,authpriv.none           -/var/log/syslog
to

.;auth,authpriv.none,local6           -/var/log/syslog
and add the following line to the end of the file:

local6.info                     /var/log/history.log

then execute:

touch /var/log/history.log && /etc/init.d/rsyslog restart

  • This question is specifically about the problem of logging ssh sessions on the side of the initiating/ local/ client computer, without having to (remember/ be allowed) to configure each remote server or having to manually download the logs from all remote servers you connect to. I think your answer, while it does not answer this question, would be useful to someone interested in improving the auditing of their server and should perhaps be moved to a more relevant question. – Oleg Jul 20 '14 at 20:53
0

How about strace -o /tmp/ssh_log -ff -s8192 -T -ttt -fp $(pidof sshd)? This logs all ssh sessions. You may need a tool to parse the log subsequently, or just use grep, awk etc.

  • -f: trace forked children
  • -ff: log each child separately to ssh_log.PID
  • -s8192: increase string logging limit (if needed)
  • -T -ttt: microsecond stamping in seconds since the Epoch
  • -p N: attach to pid N

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.