396

I have left a script running on a remote machine from when I was locally working at it. I can connect over SSH to the machine as the same user and see the script running in ps.

$ ps aux | grep ipcheck
myuser  18386  0.0  0.0  18460  3476 pts/0    S+   Dec14   1:11 /bin/bash ./ipchecker.sh

It is simply outputting to stdout on a local session (I ran ./ipchecker.sh form a local terminal window, no redirection, no use of screen etc).

Is there anyway from an SSH session I can view the output of this running command (without stopping it)?

So far the best I have found is to use strace -p 18386 but I get hordes of text flying up the screen, its far too detailed. I can stop strace and then sift through the output and find the text bring printed to stdout but its very long and confusing, and obviously whilst it's stopped I might miss something. I would like to find a way to see the script output live as if I was working locally.

Can anyone improve on this? The obvious answer is to restart the script with redirection or in a screen session etc, this isn't a mission critical script so I could do that. Rather though, I see this as a fun learning exercise.

4
  • Is your process running in a virtual console or in a GUI/xterm like environment?
    – jippie
    Dec 15, 2012 at 15:20
  • 6
    You can limit output of strace to one syscall: strace -p 4232 -e write
    – otokan
    Dec 15, 2012 at 15:59
  • @jippie The machine is running a full GUI (Linux Mynt 13, XFCE desktop), I fired up a gnome-terminal.
    – Baldrick
    Dec 15, 2012 at 16:14
  • 3
    There's at least a dozen of similar questions on this site. Look for reptyr here to find a few of them (and an answer). Dec 15, 2012 at 22:46

12 Answers 12

368

You can access the output via the proc filesystem.

tail -f /proc/<pid>/fd/1

1 = stdout, 2 = stderr

(or like @jmhostalet says: cat /proc/<pid>/fd/1 if tail doesn't work)

11
  • 13
    It gives me: 'cannot open /proc/<my pid>/fd/1 for reading: No such device or address'. Oct 27, 2016 at 13:36
  • 32
    yeah <my pid> should be your process id
    – tvlooy
    Nov 8, 2016 at 14:10
  • 68
    This won't work if the output is going to a tty (or redirected to /dev/null) — it will only work if the output is redirected to a file.
    – mattdm
    Apr 26, 2017 at 18:19
  • 3
    Tested on Ubuntu 16.04 and it doesn't work. In a session I do: ping google.es and in another one as root: tail -f /proc/`pgrep ping`/fd/2 and nothing is shown. Mar 2, 2018 at 8:57
  • 13
    tail doesn't work for me, I am using sudo cat /proc/<pid>/fd/1 instead
    – jmhostalet
    Feb 11, 2019 at 8:39
279

If all you want to do is spy on the existing process, you can use strace -p1234 -s9999 -e write where 1234 is the process ID. (-s9999 avoids having strings truncated to 32 characters, and write the system call that produces output.) If you want to view only data written on a particular file descriptor, you can use something like strace -p1234 -e trace= -e write=3 to see only data written to file descriptor 3 (-e trace= prevents the system calls from being loged). That won't give you output that's already been produced.

If the output is scrolling by too fast, you can pipe it into a pager such as less, or send it to a file with strace -o trace.log ….

With many programs, you can divert subsequent output with a ptrace hack, either to your current terminal or to a new screen session. See How can I disown a running process and associate it to a new screen shell? and other linked threads.

Note that depending on how your system is set up, you may need to run all these strace commands as root even if the process is running under your user with no extra privileges. (If the process is running as a different user or is setuid or setgid, you will need to run strace as root.) Most distributions only allow a process to trace its children (this provides a moderate security benefit — it prevents some direct malware injection, but doesn't prevent indirect injection by modifying files). This is controlled by the kernel.yama.ptrace_scome sysctl.

17
  • 24
    I don't suppose there's a way to narrow down the output to just the standard output?
    – Jonah
    Mar 28, 2013 at 0:33
  • 4
    Can you explain all the arguments?
    – User
    Mar 16, 2015 at 22:15
  • 7
    There was a lot of backslashes and numbers in a lot of the output I was getting from nodejs; Any leads on what encoding they might be in? There was plenty of plain text too, which was all i needed. Jul 16, 2015 at 17:27
  • 4
    "Can you explain all the arguments?" @User: man strace
    – Pistos
    Sep 1, 2015 at 15:32
  • 5
    @RafaelMoni A program to do what you're asking is called a debugger. Dec 22, 2015 at 12:12
17

In BSD, you can use watch which snoops a given tty, e.g.

watch /dev/pts/0

In Linux, it won't be possible if the process wasn't run under multiplexer before such as screen or tmux. See also: Reptyr: Attach a Running Process to a New Terminal

It seems the only way is to debug the process (e.g. strace, dtrace/dtruss, gdb, lldb, etc.).

Since you've used strace, to fetch any meaningful output, you need to filter by a qualifying expression (such as file), then parse the output. Here is example:

strace -e trace=write -s1000 -fp 18386 2>&1 | grep -o '".\+[^"]"'

What it does it prints write operation of the process (1000 length) specified by PID (use pgrep to find it by name), redirects standard error into output (to be filtered), and prints double-quoted string.

If you're dealing with binary output, you may parse escape sequences characters by using read (with -r) and printf (with %b), e.g.

while read -r -t1 line; do printf "%b" $line; done

Check help read for more parameters (e.g. -n to print after certain amount of characters, rather than newline).

Here is more complete example:

strace -e trace=write -s1000 -fp 18386 2>&1 \
| grep --line-buffered -o '".\+[^"]"' \
| grep --line-buffered -o '[^"]\+[^"]' \
| while read -r line; do
  printf "%b" $line;
done

For examples using any process, please check: How to parse strace in shell into plain text? at stackoverflow

9

Parsing the output of strace:

I used the top answer (with my process ID of 28223) ...

> sudo strace -p28223 -s9999 -e write
...
write(9, "Info\nI\nCare\nabout", 55) = 55
...

To determine that I care about write(9. (The 9 is used below, it's probably a file handle and might be different for your process.) Then I wrote a quick Ruby script to parse and display them.

Paste the following into /usr/bin/parse_strace.rb

#!/usr/bin/ruby

num = ARGV[0]
STDIN.each { |line|
  if (line.match(/write\(#{ num },\s*"(.*?)"/)) then
    puts $1.split('\x').map { |s| s.to_i(16).chr }.join()
  end
}

Don't forget chmod a+x /usr/bin/parse_strace.rb

I invoke strace -xx (outputs hex, so the regex matches properly), piping (including STDERR) to my script with 9 as the first arg.

sudo sh -c 'strace -xx -p28223 -s9999 -e write 2>&1 | parse_strace.rb 9'

And, voila, it outputs the process's original STDOUT, newlines, color, and all!

Attaching to process STDOUT

8

If you want to get stderr and stdout you can just run this:

tail -f /proc/<pid>/fd/*
6

You always can launch a process whith nohup and &

nohup rsync source_file dest_file &

Then, you'll can check the progress from any tty with:

tail -f nohup.out

This works fine to me.

3
  • 9
    This question I'd about how to view the output of an alert running process, not how to run a process in the background as your answer suggests
    – Baldrick
    Apr 26, 2017 at 18:18
  • 2
    In fact, your :nohup mention enlighten me! Thanks!
    – Nam G VU
    Dec 13, 2018 at 15:55
  • interesting response, would u mind to elaborate it a bit more? Jan 23, 2021 at 20:08
4
  1. You might be able to peek at the remote screen using ssh localhost 'DISPLAY=:0.0 xwd -root' | xwud -scale where localhost is to be replaced by your remote server login credentials and :0.0 with the display number of your GUI.

  2. Use x11vnc, which is a VNC server for your on screen X-session.

  3. When running on one of the 6 virtual consoles try sudo setterm -dump 2 -file /dev/stdout, where you replace 2 with the appropriate vc.

4

I'd advise to make a named pipe (mkfifo) and then write to that file. Then, read from it. You can always do that with things like tail, to minimize output, etc. Whenever you clear the pipe (read from it), it gets cleared, so the output is not preserved.

The other option would be to write everything to a file (much like a logfile) and then analyze it an any time. This would be the preferred action, if you want to preserve all output.

1

A very simple to get the output would be capturing your output to a file and tailing that file.

IF DO:

./ipcheck

INSTEAD DO:

./ipcheck > [replacewithyourfilename]

This would create an output file where your script is located. Then from any other bash shell you can simply tail the file:

tail [replacewithyourfilename] -f
2
  • File redirection tends to use block-based buffering by default (see setbuf(3)) which might make a tail problematical.
    – thrig
    Apr 26, 2017 at 18:14
  • 8
    Also this doesn't help with the question, which was about how to view the output of an already running process, not how to redirect stdout for a new process.
    – Baldrick
    Apr 26, 2017 at 18:19
1

I wrote catp to do this.

It's basically strace with only tracee's output extracted.

Usage: catp $PID

0

In bash 1:

any_command_you_want > my_output.txt&

In bash 2:

cat my_output.txt
1
  • 1
    Pleas note that the OP said "Is there anyway from an SSH session I can view the output of this running command (without stopping it)?". While your approach doesn't require stopping the command, it would not help to watch the output of a command that is already running (as in the OPs problem). Also, I would at least consider using tail -f instead of cat so that the output can be followed in near-realtime. You may want to look at the OPs comment to this answer which is basically the same as yours.
    – AdminBee
    Jul 13, 2020 at 9:19
0

If you have the commercial LSF from IBM as job scheduler, then you have also the option

bpeek displays the stdout and stderr output of an unfinished job

bpeek [-f] [-q queue_name | -m host_name | -J job_name | job_ID |
"job_ID[index_list]"]

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .