5

In mimicing MacOS's terminal's open command or Window's terminal's start, the comments of this answer suggest appending stdout and stderr to ~/.xsession-errors, something like (bash):

alias open='&>>~/.xsession-errors xdg-open'

The issue I forsee with this is race conditions. lsof ~/.xsession-errors shows 22 processes have the file open for writing.

How does one prevent two processes writing to the same offset in ~/.xsession-errors?

  • What exactly do you mean by "race conditions"? The order stdout and stderr are written? Or are you afraid both may be written "at once", and the output will be swallowed? That doesn't happen, AFAIK. – dirkt Feb 19 '17 at 7:16
  • I aim to prevent two processes both having an error to write, opening the flie, seeking to the same offset (EOF), and then writing at the same position. – Tom Hale Feb 19 '17 at 13:03
  • Hm, lsof shows that on my system, .xsession-errors is already opened by a dozen or so processes. No idea how and if they handle this ... – dirkt Feb 19 '17 at 16:14
  • @dirkt It's handled poorly based on the However section of this answer. – Tom Hale Feb 20 '17 at 7:09
7

When a file is opened in append mode, the OS guarantees that all writes take place at the end. So the data from one writer will not overwrite the data from another writer.

This only applies if the file is opened in append mode, i.e. with >> in the shell. If the creator of the file opens it with > then that guarantee won't apply and it's possible to have this sequence:

  • Process 1: >out; now at position 0
  • Process 2: >>out; now at position 0
  • Process 1: write hello, now at position 6
  • Process 2: write world, this is written at position 6 and process 2 is now at position 12
  • Process 1: write oops, this is written at position 6 because process 1's file position has not changed.

On Debian (since 2001 or thereabouts), the file .xsession-errors is created by `/etc/X11/Xsession and it is opened in append mode, so everything is fine:

 exec >>"$ERRFILE" 2>&1

I don't know if that's the case on all distributions that log to ~/.xsession-errors.

As long as everybody opens the file in append mode, all output will be present. The output may be fragmented however. In practice, small enough writes to a regular file are atomic. Anything less than 512B should be small enough everywhere, and I think Linux guarantees more than that¹. So each log line should appear intact, even with multiple concurrent writers, assuming that the writers use line-buffered output and that the lines are not overly long.

¹ Note that POSIX does not guarantee anything except for pipes.

  • POSIX guarantees a minimum write request of 32,767 bytes. See this answer. – Tom Hale Feb 20 '17 at 8:14
  • 1
    @TomHale You're misinterpreting SSIZE_MAX. POSIX guarantees that if you call write with a size <= SSIZE_MAX and it returns some size then the returned number of bytes has been returned atomically. But it's possible for a write to be partial: the program calls write(42) and write returns 32, that means that 32 bytes have been written atomically, but your 42 bytes did not return atomically. POSIX says that this is guaranteed as long as the 42 value is <= SSIZE_MAX. But write(SSIZE_MAX+1) could write two separate chunks and not tell you. – Gilles Feb 20 '17 at 9:26
  • @TomHale In practice, on Linux, you do get partial writes for sizes less than SSIZE_MAX at least on block devices. I don't know if there's a guarantee for regular files but if there is it's undocumented so it would be dependent on the kernel version. – Gilles Feb 20 '17 at 9:27
2

Using >> in a POSIX shell guarantees that the file will be opened with O_APPEND.

The Open Group Base Specifications Issue 7 states:

If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write and no intervening file modification operation shall occur between changing the file offset and the write operation.

POSIX defines the mimimum number of bytes that can be requested to be written in a single write(2) call (SSIZE_MAX = 32,767). The return value is the number of bytes actually written (guaranteed atomic).

However

Not all filesystems comply. Appending to a File from Multiple Processes says:

The caveat is that not all filesystems are POSIX-compatible. Two famous examples are NFS and the Hadoop Distributed File System (HDFS). On these networked filesystems, appends are simulated and subject to race conditions.

Not everyone plays nice

While you might open using O_APPEND, the other processes writing to the file may not. You can check for any given file:

lsof +fg <file>

Worryingly, when I run lsof +fg ~/.xsession-errors, I see no AP (append) flag, suggesting that the 22 processes in my listing (Manjaro Linux based on Arch) are not opening the file safely.

Only when I run cat >> ~/.xsession-errors in another shell does the final output line include the AP flag:

cat       3099 ravi    1w   REG   W,AP,LG   0,48      963 1926479 .xsession-errors

If anyone knows where this issue should be raised upstream, please comment.

Practically

If all processes open the file locally with either:

  • open(2) and flag O_APPEND
  • fopen(3) and the "a" flag
  • POSIX sh >> or bash &>>

Then no data should be overwritten via race conditions.

Thanks to @Gilles answer for pointing me in the right direction.

  • I have reported this for Manjaro here – Tom Hale Feb 20 '17 at 7:07
  • 1
    Note that as I commented on my answer, you're misinterpreting SSIZE_MAX: if you call write(n) with n <= SSIZE_MAX, POSIX does not guarantee that this either writes 0 or n. It only guarantees that if the function returns r then r bytes have been written atomically, but r could be less than n. – Gilles Feb 20 '17 at 9:28
  • Thanks, I've updated the text to reflect this. If you have ideas on where to best report the unsafe opening of ~/.xsession-errors, I'm all ears. – Tom Hale Feb 20 '17 at 10:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.