I am running some long-running programs (for scientific purposes) on a multi-core Linux box. The processes are controlled by a small daemon which restarts the next job when one finishes (I run 3-6 at once) - but I've noticed they don't always use 100% CPU. The programs (and the daemon) are written in Python.
When I run this code on Mac OS X, I can run programs for weeks and they will always use as many system resources are available, while the machine is running at a normal temperature.
I've just started trying to run such things on Debian Linux (on another machine), with 6 Cores and much more RAM than the jobs need. I am running 5 such jobs at once.
When I first started the jobs a few days ago, I had 5 Python processes in
top, each using 100% CPU. About a day later, I checked how it was all doing and I had 3 processes at 100% CPU and two running at 50% CPU. Most recently (about 4 days in) I have 5 processes all running at 20% CPU.
What could be causing this? Nothing seems to suggest that CPU usage management tools are pre-installed on Debian Wheezy, and I have not installed anything like that myself (to my knowledge), nor configured it. Also, since the limits seem to vary depending on how long the demon has been running, I'm not convinced that it could be such a system. I have checked if the machine is overheating, and it doesn't seem to be much hotter than the (cold) room it's in; the air from the fans/vents is both unobstructed and cool.
The processes are still running, so I can measure anything that might be useful for debugging (length of time running, process priority, etc) in order to debug this problem. Can anyone tell me where to start, or what any possible solutions might be?
When I try the same thing with 3 threads instead of 5, I am dropped to 33% each (after initial drops to 50%.)
Is there any program or scheduling policy that limits all child processes of a single process to a total of 100%? as that's what seems to be happening.
Next test is to directly run the scripts in separate
screen shells, (BTW, the first script was launched from inside
screen) and see if we get any slowdown. Carving the jobs up by hand like this is an OK-ish workaround, but quite irritating (and should be unnecessary.) In general, of course, this sort of problem may not be solvable this way, but since all results from each job are saved to disk rather than returned to the thread manager, I'll get away with it.
Separate processes launched from different screen instances are still going at 100% CPU after 14 hours, will report back if I see any slowdown but as expected this case is unaffected by any throttling.
anyone care to write (or point me at) something that explains process priority on Linux? I am wondering if my spawning processes is being marked as a lower priority (since it uses very little CPU itself), and then the child processes are inheriting this.
I've been asked about the script I'm running, and the function of forking the daemon processes.
The long running script is a big calculation, which always runs at 100% CPU until it finishes, and does nothing funny about parallelisation or multiprocessing. (this is a widely tested assertion.) To clarify further - the only times I've seen these processes run at less than 100% CPU on my Mac are when overheating, or when paging/swapping. Neither of these is relevant to the Linux case.
Here is the function that forks out, and then manages the long running processes:
from multiprocessing import Process import time, sys, os # An alternative entry point which runs N jobs in parallel over a list of files. # Note, since this is designed to be used as a library function, we "return" from the initial # function call rather than exiting. def run_over_file_list(script_path, list_of_data_files, num_processes, timeout=float('inf')): try: pid = os.fork() if pid > 0: # exit first parent return except OSError, e: print >>sys.stderr, "fork #1 failed: %d (%s)" % (e.errno, e.strerror) sys.exit(1) # decouple from parent environment os.chdir("/") os.setsid() os.umask(0) # do second fork try: pid = os.fork() if pid > 0: # exit from second parent, print eventual PID before print "Daemon PID %d" % pid sys.exit(0) except OSError, e: print >>sys.stderr, "fork #2 failed: %d (%s)" % (e.errno, e.strerror) sys.exit(1) # OK, we're inside a manager daemon. if os.path.isfile(status_filename): raise Exception("a daemon is already running. failed.") f = open(status_filename, "w") f.write(str(os.getpid())) f.close() jobs = [script_path] * num_processes data_files_remaining = [f for f in list_of_data_files] update_files_remaining_file(len(data_files_remaining)) assert num_processes <= len(data_files_remaining) restart = False with nostdout(): while True: processes =  for job in jobs: p = Process(target=file_list_worker, args=(job, data_files_remaining.pop(0))) p.started = time.time() p.start() processes.append(p) stop = False while True: time.sleep(10) ended =  for i, p in enumerate(processes): if not p.is_alive(): j = i ended.append((j,p)) elif time.time() - p.started > timeout: p.terminate() j = i ended.append((j,p)) if not stop: for tup in ended: if not data_files_remaining: stop = True break i, e = tup new_p = Process(target=file_list_worker, args=(jobs[i], data_files_remaining.pop(0))) new_p.started = time.time() new_p.start() processes[i] = new_p # old e will be garbage collected else: if len(ended) == len(processes) and not data_files_remaining: stop = False break try: command = check_for_command() if command == "stop": stop = True elif command == "restart": stop = True restart = True elif command == "kill": for p in processes: p.terminate() clear_command() os.remove(status_filename) exit(0) except NoCommandError: pass update_files_remaining_file(len(data_files_remaining)) clear_command() update_files_remaining_file(len(data_files_remaining)) if not restart: os.remove(status_filename) break else: jobs = None restart = False # While in a fork, we should never return (will continue running the original/calling script in parallel, hilarity ensues.) exit(0)
So, everything seems to run with priority
20 from whatever source; the pre-throttled processes, the post-throttled processes, the daemon manager, the processes run directly from the shell under screen.
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 127788 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 127788 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Maximum size of core files created (kB, -c) 0 Maximum size of a process’s data segment (kB, -d) unlimited Maximum size of files created by the shell (kB, -f) unlimited Maximum size that may be locked into memory (kB, -l) 64 Maximum resident set size (kB, -m) unlimited Maximum number of open file descriptors (-n) 1024 Maximum stack size (kB, -s) 8192 Maximum amount of cpu time in seconds (seconds, -t) unlimited Maximum number of processes available to a single user (-u) 127788 Maximum amount of virtual memory available to the shell (kB, -v) unlimited
from fish under screen:
(exactly the same as normal fish.)
Much later update
I have also noticed this bug with long-running processes run from separate shells. e.g:
Instance 1: 17% (one core of 6 at 100%.) Instance 2: 8% (one core of 6 at 50%.) Instance 3: 8% (one core of 6 at 50%.)
if I change the priority of instance 2 to be "very high", the state becomes:
Instance 1: 17% (one core of 6 at 100%.) Instance 2: 17% (one core of 6 at 100%.) Instance 3: 0% (one core of 6 at 0%.)
If the priorities are all equalised again, we return to the first state.
I am starting to think that this problem may be related to a specific hardware configuration or something, but I lack the tools/knowledge to debug further.