1

I am running some long-running programs (for scientific purposes) on a multi-core Linux box. The processes are controlled by a small daemon which restarts the next job when one finishes (I run 3-6 at once) - but I've noticed they don't always use 100% CPU. The programs (and the daemon) are written in Python.

When I run this code on Mac OS X, I can run programs for weeks and they will always use as many system resources are available, while the machine is running at a normal temperature.

I've just started trying to run such things on Debian Linux (on another machine), with 6 Cores and much more RAM than the jobs need. I am running 5 such jobs at once.

When I first started the jobs a few days ago, I had 5 Python processes in top, each using 100% CPU. About a day later, I checked how it was all doing and I had 3 processes at 100% CPU and two running at 50% CPU. Most recently (about 4 days in) I have 5 processes all running at 20% CPU.

What could be causing this? Nothing seems to suggest that CPU usage management tools are pre-installed on Debian Wheezy, and I have not installed anything like that myself (to my knowledge), nor configured it. Also, since the limits seem to vary depending on how long the demon has been running, I'm not convinced that it could be such a system. I have checked if the machine is overheating, and it doesn't seem to be much hotter than the (cold) room it's in; the air from the fans/vents is both unobstructed and cool.

The processes are still running, so I can measure anything that might be useful for debugging (length of time running, process priority, etc) in order to debug this problem. Can anyone tell me where to start, or what any possible solutions might be?

UPDATE:

When I try the same thing with 3 threads instead of 5, I am dropped to 33% each (after initial drops to 50%.)

Is there any program or scheduling policy that limits all child processes of a single process to a total of 100%? as that's what seems to be happening.

Next test is to directly run the scripts in separate screen shells, (BTW, the first script was launched from inside screen) and see if we get any slowdown. Carving the jobs up by hand like this is an OK-ish workaround, but quite irritating (and should be unnecessary.) In general, of course, this sort of problem may not be solvable this way, but since all results from each job are saved to disk rather than returned to the thread manager, I'll get away with it.

UPDATE 2:

Separate processes launched from different screen instances are still going at 100% CPU after 14 hours, will report back if I see any slowdown but as expected this case is unaffected by any throttling.

anyone care to write (or point me at) something that explains process priority on Linux? I am wondering if my spawning processes is being marked as a lower priority (since it uses very little CPU itself), and then the child processes are inheriting this.

EDIT:

I've been asked about the script I'm running, and the function of forking the daemon processes.

The long running script is a big calculation, which always runs at 100% CPU until it finishes, and does nothing funny about parallelisation or multiprocessing. (this is a widely tested assertion.) To clarify further - the only times I've seen these processes run at less than 100% CPU on my Mac are when overheating, or when paging/swapping. Neither of these is relevant to the Linux case.

Here is the function that forks out, and then manages the long running processes:

from multiprocessing import Process
import time, sys, os

# An alternative entry point which runs N jobs in parallel over a list of files.
# Note, since this is designed to be used as a library function, we "return" from the initial
# function call rather than exiting.
def run_over_file_list(script_path, list_of_data_files, num_processes, timeout=float('inf')):
    try:
        pid = os.fork()
        if pid > 0:
            # exit first parent
            return
    except OSError, e:
        print >>sys.stderr, "fork #1 failed: %d (%s)" % (e.errno, e.strerror)
        sys.exit(1)

    # decouple from parent environment
    os.chdir("/")
    os.setsid()
    os.umask(0)

    # do second fork
    try:
        pid = os.fork()
        if pid > 0:
            # exit from second parent, print eventual PID before
            print "Daemon PID %d" % pid
            sys.exit(0)
    except OSError, e:
        print >>sys.stderr, "fork #2 failed: %d (%s)" % (e.errno, e.strerror)
        sys.exit(1)

    # OK, we're inside a manager daemon.
    if os.path.isfile(status_filename):
        raise Exception("a daemon is already running. failed.")

    f = open(status_filename, "w")
    f.write(str(os.getpid()))
    f.close()

    jobs = [script_path] * num_processes
    data_files_remaining = [f for f in list_of_data_files]
    update_files_remaining_file(len(data_files_remaining))

    assert num_processes <= len(data_files_remaining)

    restart = False

    with nostdout():
        while True:
            processes = []

            for job in jobs:
                p = Process(target=file_list_worker, args=(job, data_files_remaining.pop(0)))
                p.started = time.time()
                p.start()
                processes.append(p)

            stop = False
            while True:
                time.sleep(10)

                ended = []
                for i, p in enumerate(processes):
                    if not p.is_alive():
                        j = i
                        ended.append((j,p))
                    elif time.time() - p.started > timeout:
                        p.terminate()
                        j = i
                        ended.append((j,p))
                if not stop:
                    for tup in ended:
                        if not data_files_remaining:
                            stop = True
                            break
                        i, e = tup
                        new_p = Process(target=file_list_worker, args=(jobs[i], data_files_remaining.pop(0)))
                        new_p.started = time.time()
                        new_p.start()
                        processes[i] = new_p
                        # old e will be garbage collected
                else:
                    if len(ended) == len(processes) and not data_files_remaining:
                        stop = False
                        break

                try:
                    command = check_for_command()
                    if command == "stop":
                        stop = True
                    elif command == "restart":
                        stop = True
                        restart = True
                    elif command == "kill":
                        for p in processes:
                            p.terminate()
                        clear_command()
                        os.remove(status_filename)
                        exit(0)
                except NoCommandError:
                    pass

                update_files_remaining_file(len(data_files_remaining))

            clear_command()

            update_files_remaining_file(len(data_files_remaining))

            if not restart:
                os.remove(status_filename)
                break
            else:
                jobs = None
                restart = False

    # While in a fork, we should never return (will continue running the original/calling script in parallel, hilarity ensues.)
    exit(0)

EDIT 2:

priority

So, everything seems to run with priority 20 from whatever source; the pre-throttled processes, the post-throttled processes, the daemon manager, the processes run directly from the shell under screen.

ulimit -a

from bash:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127788
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 127788
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

from fish:

Maximum size of core files created                           (kB, -c) 0
Maximum size of a process’s data segment                     (kB, -d) unlimited
Maximum size of files created by the shell                   (kB, -f) unlimited
Maximum size that may be locked into memory                  (kB, -l) 64
Maximum resident set size                                    (kB, -m) unlimited
Maximum number of open file descriptors                          (-n) 1024
Maximum stack size                                           (kB, -s) 8192
Maximum amount of cpu time in seconds                   (seconds, -t) unlimited
Maximum number of processes available to a single user           (-u) 127788
Maximum amount of virtual memory available to the shell      (kB, -v) unlimited

from fish under screen:

(exactly the same as normal fish.)


Much later update

I have also noticed this bug with long-running processes run from separate shells. e.g:

Instance 1: 17% (one core of 6 at 100%.)
Instance 2: 8% (one core of 6 at  50%.)
Instance 3: 8% (one core of 6 at  50%.)

if I change the priority of instance 2 to be "very high", the state becomes:

Instance 1: 17% (one core of 6 at 100%.)
Instance 2: 17% (one core of 6 at 100%.)
Instance 3: 0% (one core of 6 at    0%.)

If the priorities are all equalised again, we return to the first state.

I am starting to think that this problem may be related to a specific hardware configuration or something, but I lack the tools/knowledge to debug further.

  • To help diagnose this we're going to need to see the script, and this seems like a borderline programming Q. Focus on how you're forking the processes from Python. – slm Sep 15 '14 at 12:09
  • As I mentioned, the script works as expected on Linux to start with, and also works fine on OS X. I do not believe it is a problem with the script, which is why I've asked it here. I will post the details of the forking procedure above. – tehwalrus Sep 15 '14 at 12:15
  • 1
    Understood, but we need to see what you're doing to help debug your issue(s). Otherwise your Q is too-broad and I'd have to close it as such. – slm Sep 15 '14 at 12:17
  • It might also help to see the output of ulimit -a from the shell where you start your processes (or the equivalent limit or whatever if you're not using bash) – Useless Sep 18 '14 at 10:14
  • Oh, and if you're running top, can you see something else on CPU that would prevent your processes running? Does the priority (PR) value change over time? – Useless Sep 18 '14 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.