7

Is it OK for two or more processes concurrently read/write to the same unix socket?

I've done some testing.

Here's my sock_test.sh, which spawns 50 clients each of which concurrently write 5K messages:

#! /bin/bash --

SOC='/tmp/tst.socket'

test_fn() {
  soc=$1
  txt=$2
  for x in {1..5000}; do
    echo "${txt}" | socat - UNIX-CONNECT:"${soc}"
  done
}

for x in {01..50}; do
  test_fn "${SOC}" "Test_${x}" &
done

I then create a unix socket and capture all traffic to the file sock_test.txt:

# netcat -klU /tmp/tst.socket | tee ./sock_test.txt

Finally I run my test script (sock_test.sh) and monitor on the screen all 50 workers doing their job. At the end I check whether all messages have reached their destination:

# ./sock_test.sh
# sort ./sock_test.txt | uniq -c

To my surprise there were no errors and all 50 workers have successfully sent all 5K messages.

I suppose I must conclude that simultaneous writing to unix sockets is OK?

Was my concurrency level too low to see collisions?

Is there something wrong with my test method? How then I test it properly?

EDIT

Following the excellent answer to this question, for those more familiar with python there's my test bench:

#! /usr/bin/python3 -u
# coding: utf-8

import socket
from concurrent import futures


pow_of_two = ['B','KB','MB','GB','TB']
bytes_dict = {x: 1024**pow_of_two.index(x) for x in pow_of_two}
SOC = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
SOC.connect('/tmp/tst.socket')


def write_buffer(
    char: 'default is a' = 'a',
    sock: 'default is /tmp/tst.socket' = SOC,
    step: 'default is 8KB' = 8 * bytes_dict['KB'],
    last: 'default is 2MB' = 2 * bytes_dict['MB']):

    print('## Dumping to the socket: {0}'.format(sock))
    while True:
        in_memory = bytearray([ord(char) for x in range(step)])
        msg = 'Dumping {0} bytes of {1}'
        print(msg.format(step, char))
        sock.sendall(bytes(str(step), 'utf8') + in_memory)
        step += step
        if last % step >= last:
            break


def workers(concurrency=5):
    chars = concurrency * ['a', 'b', 'c', 'd']
    with futures.ThreadPoolExecutor() as executor:
        for c in chars:
            executor.submit(write_buffer, c)


def parser(chars, file='./sock_test.txt'):
    with open(file=file, mode='rt', buffering=8192) as f:
        digits = set(str(d) for d in range(0, 10))
        def is_digit(d):
            return d in digits
        def printer(char, size, found, junk):
            msg = 'Checking {}, Expected {:8s}, Found {:8s}, Junk {:8s}, Does Match: {}'
            print(msg.format(char, size, str(found), str(junk), size == str(found)))
        char, size, found, junk = '', '', 0, 0
        prev = None
        for x in f.read():
            if is_digit(x):
                if not is_digit(prev) and prev is not None:
                    printer(char, size, found, junk)
                    size = x
                else:
                    size += x
            else:
                if is_digit(prev):
                    char, found, junk = x, 1, 0
                else:
                    if x==char:
                        found += 1
                    else:
                        junk += 1
            prev = x
        else:
            printer(char, size, found, junk)


if __name__ == "__main__":
    workers()
    parser(['a', 'b', 'c', 'd'])

Then in the output you may observe lines like the following:

Checking b, Expected 131072  , Found 131072  , Junk 0       , Does Match: True
Checking d, Expected 262144  , Found 262144  , Junk 0       , Does Match: True
Checking b, Expected 524288  , Found 219258  , Junk 0       , Does Match: False
Checking d, Expected 524288  , Found 219258  , Junk 0       , Does Match: False
Checking c, Expected 8192    , Found 8192    , Junk 0       , Does Match: True
Checking c, Expected 16384   , Found 16384   , Junk 0       , Does Match: True
Checking c, Expected 32768   , Found 32768   , Junk 610060  , Does Match: True
Checking c, Expected 524288  , Found 524288  , Junk 0       , Does Match: True
Checking b, Expected 262144  , Found 262144  , Junk 0       , Does Match: True

You can see that payload in some cases (b, d) is incomplete, however missing fragments are received later (c). Simple math proves it:

# Expected
b + d = 524288 + 524288 = 1048576
# Found b,d + extra fragment on the other check on c
b + d + c = 219258 + 219258 + 610060 = 1048576

Therefore simultaneous writing to unix sockets is OK NOT OK.

5

That is a very short test line. Try something larger than the buffer size used by either netcat or socat, and sending that string in multiple times from the multiple test instances; here's a sender program that does that:

#!/usr/bin/env expect

package require Tcl 8.5

set socket    [lindex $argv 0]
set character [string index [lindex $argv 1] 0]
set length    [lindex $argv 2]
set repeat    [lindex $argv 3]

set fh [open "| socat - UNIX-CONNECT:$socket" w]
# avoid TCL buffering screwing with our results
chan configure $fh -buffering none

set teststr   [string repeat $character $length]

while {$repeat > 0} {
    puts -nonewline $fh $teststr
    incr repeat -1
}

And then a launcher to call that a bunch of times (25) using different test characters of great length (9999) a bunch of times (100) to hopefully blow well past any buffer boundary:

#!/bin/sh

# NOTE this is a very bad idea on a shared system
SOCKET=/tmp/blabla

for char in a b c d e f g h i j k l m n o p q r s t u v w x y; do
    ./sender -- "$SOCKET" "$char" 9999 100 &
done

wait

Hmm, I don't have a netcat hopefully nc on Centos 7 will suffice:

$ nc -klU /tmp/blabla > /tmp/out

And then elsewhere we feed data to that

$ ./launcher

Now our /tmp/out will be awkward as there are no newlines (some things buffer based on newline so newlines can influence test results if that is the case, see setbuf(3) for the potential for line-based buffering) so we need code that looks for a change of a character, and counts how long the previous sequence of identical characters was.

#include <stdio.h>

int main(int argc, char *argv[])
{
    int current, previous;
    unsigned long count = 1;

    previous = getchar();
    if (previous == EOF) return 1;

    while ((current = getchar()) != EOF) {
        if (current != previous) {
            printf("%lu %c\n", count, previous);
            count = 0;
            previous = current;
        }
        count++;
    }
    printf("%lu %c\n", count, previous);
    return 0;
}

Oh boy C! Let's compile and parse our output...

$ make parse
cc     parse.c   -o parse
$ ./parse < /tmp/out | head
49152 b
475136 a
57344 b
106496 a
49152 b
49152 a
38189 r
57344 b
57344 a
49152 b
$ 

Uh-oh. That don't look right. 9999 * 100 should be 999,900 of a single letter in a row, and instead we got...not that. a and b got started early, but it looks like r somehow got some early shots in. That's job scheduling for you. In other words, the output is corrupt. How about near the end of the file?

$ ./parse < /tmp/out | tail
8192 l
8192 v
476 d
476 g
8192 l
8192 v
8192 l
8192 v
476 l
16860 v
$ echo $((9999 * 100 / 8192))
122
$ echo $((9999 * 100 - 8192 * 122))
476
$

Looks like 8192 is the buffer size on this system. Anyways! Your test input was too short to run past buffer lengths, and gives a false impression that multiple client writes are okay. Increase the amount of data from clients and you will see mixed and therefore corrupt output.

  • I it possible that nc is mixing up the data it outputs? According to this post stackoverflow.com/questions/9644251/… it should create a separate fd on the server side for each connection and hence there should be a separate server buffer for every connection. – Artashes Aghajanyan Jan 15 at 16:42
  • @ArtashesAghajanyan the data from all those different buffers get mashed together by the shell nc ... > file redirect, and those buffers are of limited size and fill up randomly depending on when data arrives over the network. the approach is flawed and will lead to data corruption – thrig Jan 15 at 20:17
  • Just to clarify, are you suggesting that multiple clients can't connect to the same socket file and send data to a unix domain socket server in a reliable way? Because I think the author of the question made a conclusion based on your answer, which is Therefore simultaneous writing to unix sockets is NOT OK. I think ^ this statement is not accurate. You see issues not because simultaneous writing to unix sockets is not ok, but because the tools you use and the way you use them creates issues. – Artashes Aghajanyan Jan 15 at 23:07
  • Where did I suggest that? – thrig Jan 16 at 14:55
  • Sorry if I misinterpreted the conclusion of your answer. Note that the original question was Is it OK for two or more processes concurrently read/write to the same unix socket? and then the author refers to your answer to make a conclusion that Therefore simultaneous writing to unix sockets is NOT OK. I think we both agree that this conclusion is incorrect. Perhaps it's worth making it more explicit in your post that the issue is with the usage of tools such as socat and redirect. – Artashes Aghajanyan Jan 16 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.