3

I have two scripts running parallelly and they are echoing to the same file. One script is echoing +++++++++++++++ to the file while the other script is echoing =========== to the file.

Below is the first script

#!/bin/bash
while [ 1==1 ];
do
    echo "+++++++++++++++" >> log.txt
    # commands
done

Below is the second script

#!/bin/bash
while [ 1==1 ];
do
    echo "===========" >> log.txt
    # commands
done

The log.txt file has around 1400000 lines printed and not a single line has jumbled case like ++== or something like that?

Does Linux prevent this kind of jumbling from happening and if it does then how and why ?

3
  • 1
    Check this question. In a nutshell, the write will be atomic up to a certain size (PIPE_BUF). After that it might still be atomic, but it is no longer guaranteed to be. Apr 11 at 6:37
  • Also see: unix.stackexchange.com/a/452762/70524
    – muru
    Apr 11 at 7:07
  • @Eduardo Trápani, I read from what you propose, especially pubs.opengroup.org/onlinepubs/007904975/functions/write.html "Write requests of {PIPE_BUF} bytes or less shall not be interleaved with data from other processes doing writes on the same pipe.". The specification about PIPE_BUF is specific to pipes (and FIFO... some other parts of the POSIX specification deals with them)... But I havn't found something about PIPE_BUF and regular files. Apr 11 at 18:40

2 Answers 2

7

Simply, the echo command triggers one write syscall which is atomic.

Note that write doesn’t guarantee to write all bytes it is given, but in this case (few data), it does.

Then in theory write(fd, buffer, n) can write less than n bytes and return the actual written number of written bytes to enable the program to write bytes at buffer+n.

Such a thing may happen with a pipe since the pipe doesn’t have an infinite capacity.

From write(2)

If the file was open(2)ed with O_APPEND, the file offset is first set to the end of the file before writing. The adjustment of the file offset and the write operation are performed as an atomic step.

According to POSIX.1, if count is greater than SSIZE_MAX, the result is implementation-defined; see NOTES for the upper limit on Linux.

7
  • 5
    There are no reason for the write syscall to write a partial buffer, then all lines won’t be jumbled. But if you try strace cat /boot/vmlinuz* >/dev/null, you will find that the file is read and written by blocks: the whole copy is not atomic. With text outputs of a single program, fprintf can buffer the text and the whole thing send with few write syscall while executing fflush. Apr 11 at 6:19
  • 1
    The jumbling might happen if your line is longer than PIPE_BUF. Apr 11 at 6:40
  • 1
    The PIPE_BUF limit is associated with pipes. Here we have a standard file. But we can have PIPE_BUF-1 bytes written into a pipe, then afterward, trying to write 2 bytes is not atomic since the pipe is nearly full. Apr 11 at 6:59
  • 1
    @FrédéricLoyer, PIPE_BUF is not the size of the pipe, it's the size above which write()s are not guaranteed to be atomic. On Linux, PIPE_BUF is one page size, 4k, much smaller than the default size of a pipe (64k in current versions of Linux IIRC) Apr 11 at 9:39
  • 2
    This is not quite the full story: it's relevant that >> is being used here, because that causes the files to be opened with O_APPEND, which means that each write will always (atomically) write to the end of the file. In another scenario, where processes do "open file, move file pointer to end, write" (without O_APPEND), you can get the mixing effect (if one process manages to do its write between the other process setting the file pointer and writing).
    – psmears
    Apr 11 at 14:34
0

You can try out the limit with this:

#!/bin/bash                                                                                                           

outer() {
    rm log.txt
    inner() {
        a=$(perl -e 'print "'$1'"x'$2'')
        while [ 1==1 ];
        do
            echo $a >> log.txt
            # commands                                                                                                
        done
    }
    export -f inner
    timeout 5 bash -c "inner '-' $1" &
    timeout 5 bash -c "inner '+' $1" &
    wait
    sort -u log.txt
}

replace_repeats() {
    # Replace                                                                                                         
    #   ---                                                                                                           
    # with                                                                                                            
    #  -3                                                                                                             
    perl -pe 's/((.)\2+)/"$2".length$1/e'
}
# This is OK                                                                                                          
outer 10 | replace_repeats
outer 100 | replace_repeats
outer 1000 | replace_repeats
outer 4095 | replace_repeats
# This breaks.                                                                                                        
# It should give:                                                                                                     
# +4096                                                                                                               
# -4096                                                                                                               
outer 4096 | replace_repeats
3
  • But only because bash's echo, or rather stdio underneath makes more than one write() system call for large arguments. Under stdbuf -o10000, that won't break for 4096. If you wanted to check for atomicity of the write() system calls, you could use zsh's syswrite builtin (or perl's syswrite) instead of bash's echo. Apr 12 at 6:24
  • True, and with the above it will be easy to determine where the limit then is.
    – Ole Tange
    Apr 12 at 14:53
  • No as on Linux at least, the limit, if any, is going to be much larger than the limit on the size of arguments. Shells are also known to be anything but frugal with memory. Here, I'd test by mmapping large files with differing contents so as to be able to call write() with gigabytes large buffers. Apr 12 at 15:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .