Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I suspect this is not doable just because of the security implications, but here's what I'd like to do.

Basically we run a bash shell-script on our CentOS server that calls Program-A (in our case JMeter, but that is arbitrary) which runs and dumps data into a log file. After that process finishes the shell-script starts up Program-B to analyze the log.

What I would like to do is stop the instance of Program-A1 and replace it with another instance of Program-A2, or some how swap them in place so I can safely end Program-A1 without starting up Program-B prematurely.

Why would I want to do this? The main reason is that Program-A loads some configuration files at its startup and if we make changes to those config files we have to restart the program for them to take effect, which I'd like to avoid.

I understand this is most likely not possible, but if it is I would greatly appreciate the information.

EDIT: I suppose I wasn't clear enough. Basically my shellscript looks like this:

./Program-A
./Program-B

When Program-A finishes dumping to our log, Program-B picks up the log and parses it. The problem I'm having is sometimes Program-A's settings are not set correctly or something is wrong with the environment when it starts which means that we'll need to run the whole thing again. We'd like to avoid that by just replacing our first instance of Program-A (calling it Program-A1) with a new instance of Program-A (we'll call this Program-A2). Does this make more sense to everyone?

The main reason we would like to do this is because Program-A and Program-B are actually a single part of a GIANT shell script that takes hours to run. Rather than restart the whole process for one individual part, we'd like to restart the single troubled program.

share|improve this question
    
At the start of Program-A, touch a file IN_PROGRESS, and upon finish mv IN_PROGRESS to FINISHED. In the beginning of Program-B check if IN_PROGRESS exists or not. If it is, back off the current running. –  yegle Mar 1 at 0:56
    
The problem is not that I need to prevent Program-B from starting, the problem is I need to restart Program-A in place because it is part of a long series of commands that have to be run in order. Just quitting the script would cost irrecoverable hours of processing time. Thanks though, I'll keep that trick under my hat –  BCqrstoO Mar 2 at 1:55
    
You can still utilize this trick. For example, let Program-B run every minutes using crontab, but check if any Program-A has finished before running the actual code. –  yegle Mar 2 at 2:48
    
In this instance I don't think this will work. The reason being in my case this shell script has many programs it runs (both before and after) and can take several hours. So while this could work in many cases, in this instance it's not really an elegant solution for me. Thanks though. –  BCqrstoO Mar 2 at 8:54

5 Answers 5

up vote 2 down vote accepted

It looks to me as if you are trying to replace a RUNNING process from OUTSIDE the process. That's some radical stuff.

When I first looked at the question, it seemed that you are looking exactly for exec. But exec is called by the program itself. So unless you have coded the process so that you can force it to just exec another process, you can't do that while it is running.

You could potentially create a signal trap in your Program-A to exec a program with predefined name (which you can then set to be whatever you want) and then use kill on this process to force it to exec. From the outside, it will look like the process kept running - it does, it just becomes someone else. However, if you haven't done that and you want to do this on a running process, I don't think you can do anything.

However, if the outer shell is running, but it hasn't started the critical process yet, you can just replace the file of the problematic process.

share|improve this answer

One of the simplest things to do in a shell script is to run programs sequentially. You can do something like this:

if Program-A
then
    Program-B
else
    # handle problems
    if Program-A
    then
        Program-B
    else
        # Program-A failed twice in a row, get help.
    fi
fi

That kind of construct will prevent Program-B from running prematurely.

If you can change the source code to Program-A, you can take advantage of the execve() system call, which causes the kernel to overlay the program calliing execve() with the code and data from another executable file. A lot of things carry through the execve() call like open file descriptors, Process ID, but some things don't. You'll have to read up to see what modifications to make.

If I read between the lines of what you write, you may want to do a fork/other work/exec combination of system calls: the fork() would give you two processes, one of which could start Program-B, while the other goes on to execve() Program-A1. Or something. You should clarify what you want.

Further out in left field, it's possible to do a "userland exec". If all you want to do is "swap" one program for another, you can do that without an execve() system call, although the execve() is probably more efficient, and certainly less error prone.

share|improve this answer
    
Sorry, I suppose I wasn't clear enough. The shell script is calling Program-A and then Program-B sequentially, I just want to be able to restart Program-A without it moving on to Program-B. Is that more sensical? –  BCqrstoO Feb 28 at 21:14
    
@SonofLysander - see changed program fragment above. –  Bruce Ediger Feb 28 at 21:24

Maybe you can just suspend Program-A and start Program-A1, and when Program-A1 finishes, kill Program-A and resume it (at which point it will handle the SIGTERM, die, and your script will carry on with Program-B):

killall -s STOP Program-A
Program-A1
killall Program-A
killall -s CONT Program-A

If you want Program-A1 to inherit the environment from Program-A, and if on Linux (and assuming all of the environment strings of Program-A contain at least a = character as they usually do), you can do:

xargs -0 --arg-file="/proc/$(pidof Program-A
  )/environ" sh -c 'exec env -i "$@" Program-A1' sh >> file.log
share|improve this answer
    
That is actually a good idea. That might be what I'm looking for, our work schedule won't permit me from experimenting with this today but I'll give this a try –  BCqrstoO Feb 28 at 21:25
    
This won't work. The bash script is waiting for an EXIT signal from the original pid (program-A). This will get triggered upon "killall Program-A", which will trigger Program-B to start while Program-A1 is still going. –  Michael Martinez Mar 12 at 19:36
    
@MichaelMartinez, killall Program-A is run after Program-A1 has finished. And Program-A won't handle SIGTERM until it's resumed. –  Stéphane Chazelas Mar 12 at 19:54
    
@StephaneChazelas, good catch. I quickly skimmed the answer and didn't pay attention to the ordering. .... So this should work, unless Program-A holds a write lock on the log file which would prevent Program-A1 from writing to it. ... I think the better solution is the answer I give below. –  Michael Martinez Mar 12 at 20:02

The only way you can do this is to send a STOP signal to Program-A, then use gdb (Gnu debugger) to replace the suspended code with the code from Program-A1. The complexity of this is likely beyond what you're willing to invest (see http://mohit.io/blog/debugging-modifying-code-at-runtime/), so I suggest an alternative:

edit your bash script to "trap" the exit from Program-A. then prompt for manual intervention, at that point you must explicity respond "y" it is safe to proceed with Program-B. One way to do this is to put a while loop around Program -A, capture the exit code, if it is not a successful exit, then prompt for user intervention, keep looping until a valid exit is encountered. Then proceed onto Program-B.

share|improve this answer

So below you'll find this really long and probably error prone mockup of how I think such a thing might work. It inserts an execing function ab_start between program_a and program_b to ensure their respective lock files are cleaned up between calls to either function and an overlapping switch.lock is deployed for each call to ab_start. So:

  • program_a will not begin function until both its trap is set and it successfully removes switch.lock. Then it does whatever until program_b.lock shows up, all the while piping its log through sed.
  • sed monitors program_a's log output for the regex signal_output at which time it opens program_b.lock and writes the captured line to it.
    • Meanwhile though, sed is happy to also pump all log data streamed through it into program_a.log, like a smart tee.
  • program_a wraps up its last loop and tragically kills itself.
  • but program_a has trapped the USR1 signal already, so it opens the previously unevaluated /dev/fd/4 <<SUSPEND file appended to its input and executes it. It then:
    • touches switch.lock
    • & backgrounds a call to ab_start with instructions to start program_b
    • while waits for both program_b.lock and switch.lock to magically disappear at which time it...
    • touches switch.lock
    • calls ab_start with instructions to exec right back into itself.
  • meanwhile...
  • ab_start saves its exec target parameter and shifts it away, then
    • refuses to do anything but sleep or rm until both {a_b}.locks have disappeared
    • frustratedly touches {a_b}.lock
    • execs into its target
  • last...
    • program_b sets itself a trap then
    • refuses to do anything but sleep or rm until switch.lock is gone.
    • does whatever it does for however long it takes and when through
    • kills itself
    • but finds temporary reprieve in its own DIE appended here-document
    • until it successfully removes program_b.lock
    • and then it really dies
  • BUT
  • program_a then wakes up and execs itself as explained above.

    HASHBANG

    ab_start() { a_b="$1" ; shift
            while [ -e /locks/program_a.lock ] ||\
                  [ -e /locks/program_b.lock ] ; do {
                  rm /locks/program_[ab].lock
                  sleep 1
           } ; done
           touch /locks/"${a_b}.lock"
           exec "$0" "${a_b}" "$@"
    } 
    
    program_a() { trap '. /dev/fd/4' USR1 ; 
        while [ -e /locks/switch.lock ] ; do {
            rm /locks/switch.lock ; sleep 1
        } ; done
        until [ -e /locks/program_b.lock ] ; do {
            stuff...
            it...
            does...
        } ; done |\
                sed 's/signal_output/w /locks/program_b.lock'\
                    >> /logs/program_a.log
        kill -USR1 $$
    } 4<<-\SUSPEND
        touch /locks/switch.lock
        ab_start program_b "$@" &
        while [ -e /locks/switch.lock ] ||\
            [ -e /locks/program_b.lock ] ; do
            sleep 10 
        done
        touch /locks/switch.lock 
        ab_start program_a "$@"
    #   END
        SUSPEND
    
    program_b() { trap '. /dev/fd/5' INT QUIT TERM EXIT
        while [ -e /locks/switch.lock ] ; do {
            rm /locks/switch.lock ; sleep 1
        } ; done
        while $work ; do { 
            stuff ...
            it ...
            does ...
        } ; done
    } 5<<-\DIE
        while [ -e /locks/program_b.lock ] ; do {
            rm /locks/program_b.lock ; sleep 1
        } ; done
    exec 5<&0 <&-
    #   NOW
        DIE 
    

    "$@"

I'll apologize now for whatever egregious errors I may have made above, but I believe this kind of thing could work so... take it as you will. No warranty though.

Definitely at least sed I think is a good idea here.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.