1

This seems like a basic question, but I haven't been able to find it anywhere. I want to get more throughput on a memory-heavy process by running many of them on a many-core machine. These processes do not communicate with each other.

I would expect the time-to-complete for each process to be approximately independent of the number of processes running until the number of processes is close to the number of physical cores (16 in my case).

I observe that the time-to-complete gradually curves over until it is about 3 times slower for each process to run when 16 are running at the same time as when only one is running.

What's slowing them down? (More detail than the two words, "context-switching," please.) Can I do anything about this?

Edit: Michael Homer points out that I'm interested in a memory-heavy process, not a CPU-heavy one. I suppose all of those CPUs share a memory bus and that could be the bottleneck. Ideally, I'd want some sort of NUMA architecture to put the process memory "closer" to the CPUs. Does that mean I need to be looking for different hardware to solve this problem?

Here are details:

I have a simple script called quickie2.py that does some random, CPU-intensive work. I launch N of them at once with bash command lines like the following for 14 processes.

for x in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do (python quickie2.py &); done

Here are the times-to-completion for each N:

N_proc  Time to completion (sec)
1       7.29
2       7.28  7.30
3       7.27  7.28  7.38
4       7.01  7.19  7.34  7.43
5       8.41  8.94  9.51  10.27  11.73
6       7.49  7.79  7.97  10.01  10.58  10.85
7       7.71  8.72  10.22  10.43  10.81  10.81  11.42
8       10.1  10.16  10.27  10.29  10.48  10.60  10.66  10.73
9       9.94  11.20  11.27  11.35  11.61  12.43  12.46  12.99  13.53
10      9.26  12.54  12.66  12.84  12.95  13.03  13.06  13.52  13.93  13.95
11      12.46  12.48  12.65  12.74  13.69  13.92  14.14  14.39  14.40  14.69  17.13
12      13.48  13.49  13.51  13.58  13.65  13.67  14.72  14.87  14.89  14.94  15.01  15.06
13      15.47  15.51  16.72  16.79  16.79  16.91  17.00  17.45  17.75  17.78  17.86  18.14  18.48
14      15.14  15.22  16.47  16.53  16.84  17.78  18.07  19.00  19.12  19.32  19.63  19.71  19.80  19.94
15      18.05  18.18  18.58  18.69  19.84  20.70  21.82  21.93  22.13  22.44  22.63  22.81  22.92  23.23  23.23
16      20.96  21.00  21.10  21.21  22.68  22.70  22.76  22.82  24.65  24.66  25.32  25.59  26.16  26.22  26.31  26.38

Edit: Incidentally, pinning processes to cores makes the fall-off worse. See commented-out line in code listing below.

N_proc  Time to completion (sec) with CPU-pinning
1       6.95 
2       10.11  10.18 
4       19.11  19.11  19.12  19.12 
8       20.09  20.12  20.36  20.46  23.86  23.88  23.98  24.16 
16      20.24  22.10  22.22  22.24  26.54  26.61  26.64  26.73  26.75  26.78  26.78  26.79  29.41  29.73  29.78  29.90 

Here is a screenshot of htop, showing that there are indeed exactly N (14 here) cores busy:

  1  [|||||||||||||||98.0%]    5  [||              5.8%]     9  [||||||||||||||100.0%]    13 [                0.0%]
  2  [||||||||||||||100.0%]    6  [||||||||||||||100.0%]     10 [||||||||||||||100.0%]    14 [||||||||||||||100.0%]
  3  [||||||||||||||100.0%]    7  [||||||||||||||100.0%]     11 [||||||||||||||100.0%]    15 [||||||||||||||100.0%]
  4  [||||||||||||||100.0%]    8  [||||||||||||||100.0%]     12 [||||||||||||||100.0%]    16 [||||||||||||||100.0%]
  Mem[|||||||||||||||||||||||||||||||||||||3952/64420MB]     Tasks: 96, 7 thr; 15 running
  Swp[                                        0/16383MB]     Load average: 5.34 3.66 2.29 
                                                             Uptime: 76 days, 06:59:39

For completeness, here is the Python program that does some work. It only matters that it keeps the CPU busy.

# Code of quickie2.py (for completeness).

import numpy
import time

# import psutil
# psutil.Process().cpu_affinity([int(sys.argv[1])])

arena = numpy.empty(240*1024**2, dtype=numpy.uint8)

startTime = time.time()

# just do some work that takes a lot of CPU
for i in range(100):
    one = arena[:80*1024**2].view(numpy.float64)
    two = arena[80*1024**2:160*1024**2].view(numpy.float64)
    three = arena[160*1024**2:].view(numpy.float64)
    three = one + two

print(" {:.2f} ".format(time.time() - startTime))
  • 1
    How much memory does each execution of this script take? – Julie Pelletier Feb 1 '17 at 23:39
  • 1
    That script streams an awful lot of data to and from memory (and I'm pretty sure at least one of the last two assignments is a mistake, anyway). I don't think the CPU usage is the biggest factor there by a long way. Perhaps there's a better test case you could use. – Michael Homer Feb 2 '17 at 0:00
  • Each process uses 240 MB (arena) and the machine has 64420 MB, so I'm accessing about 5% of the memory on the machine. – Jim Pivarski Feb 2 '17 at 0:05
  • The statements are not a mistake, though it may look odd out of context. I allocate one big memory arena and cast portions of it as Numpy arrays so that, in the real application, I can repeatedly define large arrays and perform computations across them without reallocating them. This matters a lot in the single-threaded context. – Jim Pivarski Feb 2 '17 at 0:09
  • Maybe I've got the wrong nomenclature: the process I want to compute performs a simple operation across large arrays, so maybe I shouldn't be calling it "CPU-bound" but "memory bus-bound." (It shows up in htop as 100% CPU.) I'll change my question title. – Jim Pivarski Feb 2 '17 at 0:13
1

Now that I understand what was wrong, I know that it was a hardware limitation, not a UNIX limitation, so this isn't the appropriate place to post. However, I thought I should add some closure.

My memory-limited, independent processes were indeed running into a memory bandwidth issue. I repeated it on a Knights Landing processor and learned how to allocate Numpy arrays on its local MCDRAM. Using local memory, there was no contention on the memory bus, and the process continues to scale well above the limit I observed on normal hardware.

event rate vs number of independent processes

Here's a recipe for allocating Numpy arrays on MCDRAM, rather than normal RAM.

import ctypes
import numpy

def malloc_mcdram(size):
    libnuma = ctypes.cdll.LoadLibrary("libnuma.so")
    assert libnuma.numa_available() == 0   # NUMA not available is -1

    libnuma.numa_alloc_onnode.restype = ctypes.POINTER(ctypes.c_uint8)
    return libnuma.numa_alloc_onnode(ctypes.c_size_t(size), ctypes.c_int(1))

def custom_allocator_array(allocator, size, dtype):
    ptr = allocator(size)
    ptr.__array_interface__ = {"version": 3,
                               "typestr": numpy.ctypeslib._dtype(type(ptr.contents)).str,
                               "data": (ctypes.addressof(ptr.contents), False),
                               "shape": (size,)}
    return numpy.array(ptr, copy=False).view(dtype)

myarray = custom_allocator_array(malloc_mcdram, sizeInBytes, numpy.float64)
0

You process is memory heavy, not cpu heavy. Try this instead:

#!/usr/bin/env python

import datetime
import hashlib

data = "\0" * 64

ts_start = datetime.datetime.now()
for i in range(10000000):
    data = hashlib.sha512(data).digest()
ts_end = datetime.datetime.now()
print("Elapsed: %s" % (ts_end - ts_start))

I'm getting consistent results, ca 20s to complete, on my 2-sockets / 8-cores / 16-threads machine when running up to 8 runs in parallel. Over that the performance drops as the processes start to fight over the CPU resources.

Single run:

~$ python cpuheavy.py 
Elapsed: 0:00:20.461652

8 in parallel (= 1 for each core), still the same time:

~$ for i in $(seq 8); do python cpuheavy.py & done
Elapsed: 0:00:18.979012
Elapsed: 0:00:19.092770
Elapsed: 0:00:19.873763
Elapsed: 0:00:20.139105
Elapsed: 0:00:20.147066
Elapsed: 0:00:20.181319
Elapsed: 0:00:21.328754
Elapsed: 0:00:21.495310

With 16 runs in parallel (= 1 for each hyperthread) the time increased to ca 31s as the processes started to fight over the cpu time. Ca 50% increase in time.

With 32 runs in parallel it went down the hill as the processes had to share the cpu threads. Time to complete increased to over 2 minutes for each process (4x increase in time).

  • I'm sorry--- I used the wrong words when I first posed this question and I hoped I had changed it in time. Yes, my process is memory-heavy, not CPU-heavy. I want to know why scaling fails for a memory-heavy process and if there's a way to fix that. Changing the test to one that doesn't resemble my problem helps to clarify that point but it doesn't solve my actual problem. Does anyone know about bottlenecks in memory-heavy processes? – Jim Pivarski Feb 2 '17 at 10:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.