3

This seems like a common enough problem that it's probably a duplicate, but I could not find any other similar question.

I have something like a log file that I append to, but I also want to keep it sorted, so I have something like this:

echo "foo" >> file.txt
LC_ALL=C sort -k1,1 -u -o file.txt file.txt

It's a really small file (<1000 lines), so the writing and sorting should be very quick. However, sometimes I have maybe 5 events per second, but some of them get lost. I was very surprised to see this problem with such low throughput application.

Appending to file should work every time, so it's probably the sorting where the problem happens. I thought -o (write result to FILE instead of standard output) would take care of the concurrency issues, but maybe not.

Is there a proper way of sorting a file if a different process might be writing to it?

2
  • 1
    I'm thinking you're going to need a sort implementation designed with this in mind. Should be doable, since sorting data doesn't change its size... But maybe you really ought to be using some other log method?
    – derobert
    Commented May 5, 2017 at 18:53
  • @derobert Yes, maybe this approach is just a bad idea. I just couldn't think of anything better.
    – burger
    Commented May 5, 2017 at 19:20

1 Answer 1

0

Looking at the sort implementation I have the easiest access to (OpenBSD's), it looks like you would lose data if the input file was appended to in-between

  • the sort utility having finished reading the data from the input file and commencing the actual sorting, and
  • the sort utility renaming its temporary output file to the name of the real output file (given by the -o argument) just before exiting.

I can't really see a way of preventing this other than temporarily halting any process writing to the file before sorting it, and then starting them again when the sorting was done. This could be done with kill -STOP and kill -CONT respectively.

The process writing to the file will need to open it for writing again as it would otherwise write to a file that is no longer in the directory structure (since it has been replaced). GNU sort seems to replace the contents of the original file with the sorted data, so this is not an issue there.

1
  • That assumes you can find every process that will be writing to the file. That sounds not trivial.
    – burger
    Commented May 6, 2017 at 21:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .