8

Is anyone aware of throughput benchmarks / measurements for using a local unix socket for inter-process communication?

I want to illustrate the performance benefit of having a local database instance on the same server as the software that is requesting the data from the database vs. having to communicate over a network link, especially one like gigabit Ethernet which I expect to be rather slow relatively speaking.

When searching online I found some benchmarks showing number of operations per second, but not throughput per second (i.e. 12GB/s).

I understand that the performance will vary due to things such as perhaps memory throughput on a given system or other hardware characteristics, but just a rough idea is needed.

This is not referring to local TCP performance or a comparison to that.

  • You are, actually, referring to local vs. network TCP performance. It's also the wrong thing to measure in your scenario. – Satō Katsura Nov 28 '16 at 9:49
  • @SatoKatsura I'm referring to en.wikipedia.org/wiki/Unix_domain_socket – sa289 Nov 28 '16 at 9:54
  • Yup. And how do you think UNIX domain sockets are actually implemented? – Satō Katsura Nov 28 '16 at 10:01
  • @SatoKatsura Not certain, but there's some difference based on what I've read even if it's not night and day different. Also there are benchmarks comparing local unix domain sockets to local TCP sockets which show a significant performance difference. – sa289 Nov 28 '16 at 10:22
  • Also there are benchmarks comparing local unix domain sockets to local TCP sockets which show a significant performance difference. - Can you please point to one such benchmark? – Satō Katsura Nov 28 '16 at 10:28
14
+25

You can use socat for a simple UNIX socket speed test.

Below are the results I get on my laptop:

#Generate 1GB random file in the "shared memory" (i.e. RAM disk) 
>dd if=/dev/urandom of=/dev/shm/data.dump bs=1M count=1024

Memory to disk (SSD), through UNIX socket

>socat -u -b32768 UNIX-LISTEN:/tmp/unix.sock ./data.dump &
>socat -u -b32768 "SYSTEM:dd if=/dev/shm/data.dump bs=1M count=1024" UNIX:/tmp/unix.sock
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 1.96942 s, 545 MB/s

Memory to memory, through UNIX socket

>socat -u -b32768 UNIX-LISTEN:/tmp/unix.sock /dev/shm/data.dump.out &
>socat -u -b32768 "SYSTEM:dd if=/dev/shm/data.dump bs=1M count=1024" UNIX:/tmp/unix.sock
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.927163 s, 1.2 GB/s

Memory to /dev/null (discard), through UNIX socket

>socat -u -b32768 UNIX-LISTEN:/tmp/unix.sock /dev/null &
>socat -u -b32768 "SYSTEM:dd if=/dev/shm/data.dump bs=1M count=1024" UNIX:/tmp/unix.sock
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.720415 s, 1.5 GB/s

/dev/zero to /dev/null, through UNIX socket

>socat -u -b32768 UNIX-LISTEN:/tmp/unix.sock /dev/null &
>socat -u -b32768 "SYSTEM:dd if=/dev/zero bs=1M count=1024" UNIX:/tmp/unix.sock
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.491179 s, 2.2 GB/s

As you can see even "memory to disk" test throughput is 545MB/s (i.e. ~ 4360MiB/s), which is far ahead of a maximum theoretical throughput for the 1GB ethernet connection (which is ~ 1000/8 = 125MB/s, not even considering any protocol overheads).

P.S.

Please note that this is just a simple test using some simple tools, and not a real, proper benchmark.

  • 1
    As I say below - do not confuse bandwidth with throughput. socat will tell you bandwidth under "ideal" conditions, will tell you if the theoritical bandwith is not being reached - but it will not tell you anything about delays causing application slow-down. Compare disk i/o at 8Gbit - stress test. That is the maximum you can get - whatever X is. If the application reaches that "the media" may be your bottleneck. If the application does not reach that level - the "bottleneck" is not the media. If socat maxes at 1Gbit, but the app does not - socat does not tell me what is limiting "throughput". – Michael Felt Dec 14 '16 at 22:35
3

My "answer" is lengthy - they key is to not confuse 'throughput' with 'bandwidth' - although 'bandwidth' can be a limiting factor

In short, your throughput may be limited even though your bandwidth is not saturated.


I have had to help people understand the impact of multi-tier application stacks.

For the aspect of TCP communications I make use of differences in RTT (round-trip-time).

For single-tier you can compare the local IP address (on a NIC) with lo0 (loopback).

For multi-tier you compare/compute the "more-distant" addresses, e.g., multi-tier can be either two VM in the same host, or it could be different hosts in the same datacenter, or they could be in different data centers (maybe only 500 meters distance, but still different).

FYI: for many applications RTT differences are negligable, but for applications that do 10-100 of thousands of small messages for application RTT time can become a bottleneck.

(I have seen situaltions where the "batch took nearly 6 hours longer in multi-tier when the RTT was .25 millisec longer, compared to single-tier)

So, simple test bed:

The

for host in 127.0.0.1 192.168.129.63 192.168.129.72 192.168.129.254 192.168.129.71 p5.aixtools.net
do
    wget -q http://${host}/ -O - >/dev/null
    sleep 1
done

And my monitoring program is tcpdump - with the option -ttt

   -ttt
        Prints a delta (in microseconds) between current and previous line on each dump line.

A microsecond is an SI unit of time equal to one millionth (0.000001 or 10−6 or 1/1,000,000). That is, 1000 microseconds == 1 millisecond.

So, in two different windows I have tcpdump running:

For the "local" times: tcpdump -i lo0 -n -ttt port 80 And for the "remote" tcpdump -I en1 -n -ttt port 80

In the data below - the goal is not to do any analysis, but to show how you can identify 'differences' in time required for transactions to complete. When an application throughput is serial transactions - the throughput per "sec|min|hour" is affected by the total time required for "responses". I have found this easiest to explain using the concept of RTT - round-trip-time.

For a real analysis there are additional things to be looking at. So, the only lines I shall show are the initial TCP handshake, and the first outgoing packet and the returning ACK. For the comparison compare the delta times of how long before the "reply" comes back.

127.0.0.1

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type 0, capture size 96 bytes
00:00:00.000000 IP 127.0.0.1.42445 > 127.0.0.1.80: S 1760726915:1760726915(0) win 65535 <mss 16856,nop,wscale 2,nop,nop,timestamp 1482096651 0>
00:00:00.**000035** IP 127.0.0.1.80 > 127.0.0.1.42445: S 3339083773:3339083773(0) ack 1760726916 win 65535 <mss 16856,nop,wscale 2,nop,nop,timestamp 1482096651 1482096651>
00:00:00.000013 IP 127.0.0.1.42445 > 127.0.0.1.80: . ack 1 win 33688 <nop,nop,timestamp 1482096651 1482096651>
00:00:00.**000014** IP 127.0.0.1.80 > 127.0.0.1.42445: . ack 1 win 33688 <nop,nop,timestamp 1482096651 1482096651>

192.168.129.63

note the 01.XXXXXX - for the one second sleep on the "lo0" interface

00:00:01.006055 IP 192.168.129.63.42446 > 192.168.129.63.80: S 617235346:617235346(0) win 65535 <mss 16856,nop,wscale 2,nop,nop,timestamp 1482096653 0>
00:00:00.**000032** IP 192.168.129.63.80 > 192.168.129.63.42446: S 1228444163:1228444163(0) ack 617235347 win 65535 <mss 16856,nop,wscale 2,nop,nop,timestamp 1482096653 1482096653>
00:00:00.000014 IP 192.168.129.63.42446 > 192.168.129.63.80: . ack 1 win 33688 <nop,nop,timestamp 1482096653 1482096653>
00:00:00.**000010** IP 192.168.129.63.80 > 192.168.129.63.42446: . ack 1 win 33688 <nop,nop,timestamp 1482096653 1482096653>

192.168.129.72

virtual machine in same host - note the time starts at 00.000000 - first packet displayed (and the 01.XXXXXX for the other two addresses below)

root@x063:[/]tcpdump -i en1 -n -ttt port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en1, link-type 1, capture size 96 bytes
00:00:00.000000 IP 192.168.129.63.42447 > 192.168.129.72.80: S 865313265:865313265(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 1482096655 0>
00:00:00.**000125** IP 192.168.129.72.80 > 192.168.129.63.42447: S 916041515:916041515(0) ack 865313266 win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 1481318272 1482096655>
00:00:00.000028 IP 192.168.129.63.42447 > 192.168.129.72.80: . ack 1 win 32761 <nop,nop,timestamp 1482096655 1481318272>
00:00:00.**000055** IP 192.168.129.72.80 > 192.168.129.63.42447: . ack 1 win 65522 <nop,nop,timestamp 1481318272 1482096655>

192.168.129.254

my router - outside of the host, not a virtual machine.

00:00:01.005947 IP 192.168.129.63.42448 > 192.168.129.254.80: S 2756186848:2756186848(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 1482096657 0>
00:00:00.**000335** IP 192.168.129.254.80 > 192.168.129.63.42448: S 2327415811:2327415811(0) ack 2756186849 win 5792 <mss 1460,nop,nop,timestamp 44854195 1482096657,nop,wscale 2,nop,opt-14:03>
00:00:00.000022 IP 192.168.129.63.42448 > 192.168.129.254.80: . ack 1 win 32761 <nop,nop,timestamp 1482096657 44854195>
00:00:00.**000090** IP 192.168.129.63.42448 > 192.168.129.254.80: P 1:142(141) ack 1 win 32761 <nop,nop,timestamp 1482096657 44854195>

192.168.129.71

same connection as 192.168.129.72, but this is 'busy' while '72' is idle. I would hope that the initial handshakes are nearly identical

00:00:01.005093 IP 192.168.129.63.42449 > 192.168.129.71.80: S 249227688:249227688(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 1482096659 0>
00:00:00.**000072** IP 192.168.129.71.80 > 192.168.129.63.42449: S 1898177685:1898177685(0) ack 249227689 win 65535 <mss 1460,nop,wscale 2,nop,nop,timestamp 1482096104 1482096659>
00:00:00.000022 IP 192.168.129.63.42449 > 192.168.129.71.80: . ack 1 win 32761 <nop,nop,timestamp 1482096659 1482096104>
00:00:00.**000050** IP 192.168.129.71.80 > 192.168.129.63.42449: . ack 1 win 65522 <nop,nop,timestamp 1482096104 1482096659>

multiple hops

this is the same host, same apache result, but now via the external interface (6 IP hops, rather than direct) - now you can the effect of long-distance RTT. (p.s., I modified the IP address slightly). More important - notice that there are two outgoing packets after the initial handshake before the first ACK after a handshake returns.

So, rather than 25msec RTT, think that the RTT is 250 microseconds, compared to 25 microseconds - and you have 500k transactions (that comes to only 120 to 125 seconds extra compared to local, and the throughput is, imho, comparable. But with 50M transactions (as I had in a real life situation) you gain an additional 12500 seconds - which adds about 3.5 additional hours for "literally" the same job. (and part of the solution for this case was to make the packets larger - the average size was originally 400-450 bytes).

Recall, what I want to show here is a fairly simple way to account for differences in overall time for an application (batch job) to complete when comparing multi-tier with single-tier architectures.

00:00:01.162974 IP 192.168.129.63.42450 > XX.85.86.223.80: S 1331737569:1331737569(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 1482096661 0>
00:00:00.**023962** IP XX.85.86.223.80 > 192.168.129.63.42450: S 3130510306:3130510306(0) ack 1331737570 win 65535 mss 1460,nop,wscale 2,nop,nop,timestamp 1482096106 1482096661,nop,opt-14:03>
00:00:00.000025 IP 192.168.129.63.42450 > XX.85.86.223.80: . ack 1 win 32761 <nop,nop,timestamp 1482096661 1482096106>
00:00:00.000062 IP 192.168.129.63.42450 > XX.85.86.223.80: P 1:142(141) ack 1 win 32761 <nop,nop,timestamp 1482096661 1482096106>
00:00:00.**024014** IP XX.85.86.223.80 > 192.168.129.63.42450: . ack 1 win 65522 <nop,nop,timestamp 1482096107 1482096661>

Another thing I "like" about using tcpdump is that is a program that is generally available. Nothing extra needs to be installed.

  • 1
    and how does that have anything to do with unix sockets? – nonchip Nov 1 '17 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.