2

How can I set up a process that has had its output redirected via process substitution so that it can survive after the parent script exits?

I am trying to set up a Bash script to monitor the output of a process when it starts up and then release that process so that it can continue normally (I want to call this script in other scripts and have it block further execution until start up is complete).

Right now, I am using the http.server module as my test case. The code below properly starts up the Python module and checks its output looking for a GET request and then exits. However, when it exits, the http.server process also dies. How can I make it so that this process survives after bash exits?

exec 3< <(python3 -u -m http.server 2>&1 )
while true; do
        read -t 2 line
        echo "reading..."
        echo "$line"
        if [[ "$line" =~ .*GET.* ]]; then
            break
        fi
done <&3

I thought the process was dying because the file descriptor 3 is closed when bash exits and Python is exiting when it sees its output file closed. I have tried various redirections of 3, but they didn't help. Maybe something else about the process needs to be modified so that it will survive? Basically I want the equivalent of executing python3 -u -m http.server 2>&1 & but with the ability to monitor the process's output temporarily when it first starts.

  • How are you invoking the script? From a command line, from a cron job, from another script (invoked from where), …? – Gilles Jun 10 '15 at 21:08
  • So far I have been testing by invoking the script from the command line. My plan is to invoke the script with the system/child_subprocess module of the Firefox Add-on SDK. When I try to visit a certain address in Firefox, the add-on will start the necessary process (a web server or ssh tunnel) if it is not running already and block the page-load until it is ready (or until the process fails/times out). – ws_e_c421 Jun 10 '15 at 21:15
  • The process is not killed when the shell exits, so the reason for it dying must be something else that happens afterwards. If you run it from a terminal, closing the terminal might do it. – Gilles Jun 10 '15 at 21:22
  • It does seem like if I check with pgrep right after making a GET request to the server that http.server is still running but it is gone when I check again shortly after. What happens to file descriptor 3 when the script exits? Perhaps file descriptor 3 is closed after the script exits and when Python notices that its stdout and stderr are redirected to a closed file it also exits? – ws_e_c421 Jun 10 '15 at 21:27
  • Why don't you open that pipe in a process context that will survive it? It's probably not a good idea to leave orphaned processes in the background writing to random file descriptors for what appears to be no good reason. Do what Gilles says and set up a log. – mikeserv Jun 11 '15 at 1:13
2

From the bash script, you start a subprocess (running a Python program), and create a pipe from it to the bash script.

After the bash script exit, there is no longer any process that has the pipe open for reading. Therefore the next time the Python script writes something to the pipe, it receives a SIGPIPE signal and dies. You would see the resulting error if you'd only redirected standard output to the pipe and not standard error as well.

Even if the Python script ignored the SIGPIPE signal, it would still die when writing to the pipe because the write to the pipe would fail, which would trigger an exception.

If you only want to see the line that tells you the server is ready, or up to the line about the first request, a simple way is to keep a reader running:

exec 3< <(python3 -u -m http.server 2>&1 )
{ while true; do
  …
  done;
  exec cat;
} <&3

You may want to redirect standard error to a log file somewhere rather than blissfully ignoring error messages.

An alternative approach would be to write a few lines of Python, overriding the BaseHTTPRequestHandler.log_request method to emit a log entry the first time only.

  • Your suggestion allows the loop to be broken when a match occurs and continues to print to stdout, but the exec cat command never exits until Python does. I want to somehow background Python so it survives the script because I want to block page loading until the process is ready. I thought redirecting 3 to /dev/null would help but it didn't. Don't worry about stderr. Maybe http.server is a bad test process since it uses stderr for real output. I am more likely to use this script with IPython Notebook and ssh tunnels to servers running Trac or Gollum with only port 22 open to the public. – ws_e_c421 Jun 11 '15 at 2:22
  • @ws_e_c421 Yes, cat doesn't exit until Python does, that's the whole point. Something needs to keep reading Python's output. The alternative is modifying the Python program to stop writing to stdout before the reader goes away. – Gilles Jun 11 '15 at 6:45
  • The current answer doesn't meet my requirements (script has to exit after the server process is initialized and has to leave the server process running). Your suggestion of using a log file does what I want though: use python -u -m http.server &> out.log & to start the server and then cat out.log in the loop rather than using read. The only downside is that it leaves the process writing to a log file I don't need, but as long as it doesn't grow too big that should be okay. The purpose of my question was to ask how to do this without using a log file. – ws_e_c421 Jun 12 '15 at 12:03
  • These are the observations that led to the question: 1. A process started with & at the end survives the script and still prints to the terminal. 2. A process with output redirected to /dev/null runs normally without pipe failure. So I thought I should be able somehow to start a background process redirected temporarily to 3, read its output for some time, and then redirect that output to /dev/null and forget about it. – ws_e_c421 Jun 12 '15 at 12:07
  • @ws_e_c421 You have two sensible options: change the python script so that it stops writing, or keep the stdout descriptor working (e.g. make it a file, or keep a process listening on the pipe). (There's a third option which is to use a debugger to cause the program to close the pipe and open /dev/null on the same descriptor, but it is not a sensible option, it may crash the program, it won't always work because of security restrictions, …) If your requirements don't allow either of the solutions, you're going to need to change the requirements. – Gilles Jun 12 '15 at 12:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.