197

I have left a script running on a remote machine from when I was locally working at it. I can connect over SSH to the machine as the same user and see the script running in ps.

$ ps aux | grep ipcheck
myuser  18386  0.0  0.0  18460  3476 pts/0    S+   Dec14   1:11 /bin/bash ./ipchecker.sh

It is simply outputting to stdout on a local session (I ran ./ipchecker.sh form a local terminal window, no redirection, no use of screen etc).

Is there anyway from an SSH session I can view the output of this running command (without stopping it)?

So far the best I have found is to use strace -p 18386 but I get hordes of text flying up the screen, its far too detailed. I can stop strace and then sift through the output and find the text bring printed to stdout but its very long and confusing, and obviously whilst it's stopped I might miss something. I would like to find a way to see the script output live as if I was working locally.

Can anyone improve on this? The obvious answer is to restart the script with redirection or in a screen session etc, this isn't a mission critical script so I could do that. Rather though, I see this as a fun learning exercise.

  • Is your process running in a virtual console or in a GUI/xterm like environment? – jippie Dec 15 '12 at 15:20
  • 4
    You can limit output of strace to one syscall: strace -p 4232 -e write – otokan Dec 15 '12 at 15:59
  • @jippie The machine is running a full GUI (Linux Mynt 13, XFCE desktop), I fired up a gnome-terminal. – jwbensley Dec 15 '12 at 16:14
  • 3
    There's at least a dozen of similar questions on this site. Look for reptyr here to find a few of them (and an answer). – Stéphane Chazelas Dec 15 '12 at 22:46
179

If all you want to do is spy on the existing process, you can use strace -p1234 -s9999 -e write where 1234 is the process ID. (-s9999 avoids having strings truncated to 32 characters, and write the system call that produces output.) If you want to view only data written on a particular file descriptor, you can use something like strace -p1234 -e trace= -e write=3 to see only data written to file descriptor 3 (-e trace= prevents the system calls from being loged). That won't give you output that's already been produced.

If the output is scrolling by too fast, you can pipe it into a pager such as less, or send it to a file with strace -o trace.log ….

With many programs, you can divert subsequent output with a ptrace hack, either to your current terminal or to a new screen session. See How can I disown a running process and associate it to a new screen shell? and other linked threads.

Note that depending on how your system is set up, you may need to run all these strace commands as root even if the process is running under your user with no extra privileges. (If the process is running as a different user or is setuid or setgid, you will need to run strace as root.) Most distributions only allow a process to trace its children (this provides a moderate security benefit — it prevents some direct malware injection, but doesn't prevent indirect injection by modifying files). This is controlled by the kernel.yama.ptrace_scome sysctl.

  • 17
    I don't suppose there's a way to narrow down the output to just the standard output? – Jonah Mar 28 '13 at 0:33
  • 3
    Can you explain all the arguments? – User Mar 16 '15 at 22:15
  • 6
    There was a lot of backslashes and numbers in a lot of the output I was getting from nodejs; Any leads on what encoding they might be in? There was plenty of plain text too, which was all i needed. – ThorSummoner Jul 16 '15 at 17:27
  • 3
    "Can you explain all the arguments?" @User: man strace – Pistos Sep 1 '15 at 15:32
  • 3
    @RafaelMoni A program to do what you're asking is called a debugger. – Gilles Dec 22 '15 at 12:12
138

You can access the output via the proc filesystem.

tail -f /proc/<pid>/fd/1

1 = stdout, 2 = stderr

  • 7
    It gives me: 'cannot open /proc/<my pid>/fd/1 for reading: No such device or address'. – Yaroslav Nikitenko Oct 27 '16 at 13:36
  • 17
    yeah <my pid> should be your process id – tvlooy Nov 8 '16 at 14:10
  • 29
    This won't work if the output is going to a tty (or redirected to /dev/null) — it will only work if the output is redirected to a file. – mattdm Apr 26 '17 at 18:19
  • 1
    Tested on Ubuntu 16.04 and it doesn't work. In a session I do: ping google.es and in another one as root: tail -f /proc/`pgrep ping`/fd/2 and nothing is shown. – david.perez Mar 2 '18 at 8:57
  • 1
    you are right. Because fd 1 and 2 are symlinks to the /dev/pts device. You can't tail -f a pts device. Schedule your ping command as a cronjob and then do the same. tail -f will then work – tvlooy Mar 2 '18 at 11:09
9

In BSD, you can use watch which snoops a given tty, e.g.

watch /dev/pts/0

In Linux, it won't be possible if the process wasn't run under multiplexer before such as screen or tmux. See also: Reptyr: Attach a Running Process to a New Terminal

It seems the only way is to debug the process (e.g. strace, dtrace/dtruss, gdb, lldb, etc.).

Since you've used strace, to fetch any meaningful output, you need to filter by a qualifying expression (such as file), then parse the output. Here is example:

strace -e trace=write -s1000 -fp 18386 2>&1 | grep -o '".\+[^"]"'

What it does it prints write operation of the process (1000 length) specified by PID (use pgrep to find it by name), redirects standard error into output (to be filtered), and prints double-quoted string.

If you're dealing with binary output, you may parse escape sequences characters by using read (with -r) and printf (with %b), e.g.

while read -r -t1 line; do printf "%b" $line; done

Check help read for more parameters (e.g. -n to print after certain amount of characters, rather than newline).

Here is more complete example:

strace -e trace=write -s1000 -fp 18386 2>&1 \
| grep --line-buffered -o '".\+[^"]"' \
| grep --line-buffered -o '[^"]\+[^"]' \
| while read -r line; do
  printf "%b" $line;
done

For examples using any process, please check: How to parse strace in shell into plain text? at stackoverflow

4
  1. You might be able to peek at the remote screen using ssh localhost 'DISPLAY=:0.0 xwd -root' | xwud -scale where localhost is to be replaced by your remote server login credentials and :0.0 with the display number of your GUI.

  2. Use x11vnc, which is a VNC server for your on screen X-session.

  3. When running on one of the 6 virtual consoles try sudo setterm -dump 2 -file /dev/stdout, where you replace 2 with the appropriate vc.

4

I'd advise to make a named pipe (mkfifo) and then write to that file. Then, read from it. You can always do that with things like tail, to minimize output, etc. Whenever you clear the pipe (read from it), it gets cleared, so the output is not preserved.

The other option would be to write everything to a file (much like a logfile) and then analyze it an any time. This would be the preferred action, if you want to preserve all output.

3

You always can launch a process whith nohup and &

nohup rsync source_file dest_file &

Then, you'll can check the progress from any tty with:

tail -f nohup.out

This works fine to me.

  • 6
    This question I'd about how to view the output of an alert running process, not how to run a process in the background as your answer suggests – jwbensley Apr 26 '17 at 18:18
  • In fact, your :nohup mention enlighten me! Thanks! – Nam G VU Dec 13 '18 at 15:55
1

A very simple to get the output would be capturing your output to a file and tailing that file.

IF DO: ./ipcheck

INSTEAD DO: ./ipcheck > [replacewithyourfilename]

This would create an output file where your script is located. Then from any other bash shell you can simply tail the file:

tail [replacewithyourfilename] -f

  • File redirection tends to use block-based buffering by default (see setbuf(3)) which might make a tail problematical. – thrig Apr 26 '17 at 18:14
  • 5
    Also this doesn't help with the question, which was about how to view the output of an already running process, not how to redirect stdout for a new process. – jwbensley Apr 26 '17 at 18:19
1

Parsing the output of strace:

I used the top answer (with my process ID of 28223) ...

> sudo strace -p28223 -s9999 -e write
...
write(9, "Info\nI\nCare\nabout", 55) = 55
...

To determine that I care about write(9. (The 9 is used below, it's probably a file handle and might be different for your process.) Then I wrote a quick Ruby script to parse and display them.

Paste the following into /usr/bin/parse_strace.rb

#!/usr/bin/ruby

num = ARGV[0]
STDIN.each { |line|
  if (line.match(/write\(#{ num },\s*"(.*?)"/)) then
    puts $1.split('\x').map { |s| s.to_i(16).chr }.join()
  end
}

Don't forget chmod a+x /usr/bin/parse_strace.rb

I invoke strace -xx (outputs hex, so the regex matches properly), piping (including STDERR) to my script with 9 as the first arg.

sudo sh -c 'strace -xx -p28223 -s9999 -e write 2>&1 | parse_strace.rb 9'

And, voila, it outputs the process's original STDOUT, newlines, color, and all!

Attaching to process STDOUT

0

can't you get the process ID and communicate with it with USR1,

$pgrep -l '^ipchecker.sh$'

which prints your script's PID, then use it to

$ kill -USR1 PID

I understand that USR1 is a "user defined" signal, meaning that whoever created the program can use it to mean "shut down" or "dump your logs" or "print foo a thousand times" or whatever.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.