10

I want to move large file created by external process as soon as it's closed.

Is this test command correct?

if lsof "/file/name"
then
        # file is open, don't touch it!
else
        if [ 1 -eq $? ]
        then
                # file is closed
                mv /file/name /other/file/name
        else
                # lsof failed for some other reason
        fi
fi

EDIT: the file represents a dataset and I have to wait until it's complete to move it so another program can act on it. That's why I need to know if the external process is done with the file.

5
  • 3
    Side note: once a file is opened, processes use file descriptors and inode data to manipulate it. Changing the path (i.e. moving the file) won't cause too much trouble to the process. – John WH Smith Dec 29 '14 at 9:47
  • 2
    Do you have any control over the external process? Would it be possible for the external process to create a temporary file and rename the file once it's finished writing to it? – Jenny D Dec 29 '14 at 11:33
  • @JennyD I did some investigation and it turns out to be true. I don't need lsof at all I just need to check if the file extension is not .tmp. That makes it trivial. However I'm glad I asked my question since I learned a bit about lsof and inotify and stuff. – Peter Kovac Dec 29 '14 at 11:59
  • @PeterKovac I learned more about them too, from reading the answers, so I'm very glad you asked it. – Jenny D Dec 29 '14 at 12:48
  • @JohnWHSmith - That's normally true if moving the file within the same filesystem, if he moves the file to a new filesystem before the writer has finished writing to it, he'll lose some data. – Johnny Dec 29 '14 at 20:01
11

From the lsof man page

Lsof returns a one (1) if any error was detected, including the failure to locate command names, file names, Internet addresses or files, login names, NFS files, PIDs, PGIDs, or UIDs it was asked to list. If the -V option is specified, lsof will indicate the search items it failed to list.

So that would suggest that your lsof failed for some other reason clause would never be executed.

Have you tried just moving the file while your external process still has it open? If the destination directory is on the same filesystem, then there should be no problems with doing that unless you need to access it under the original path from a third process as the underlying inode will remain the same. Otherwise I think mv will fail anyway.

If you really need to wait until your external process is finished with the file, you are better to use a command that blocks instead of repeatedly polling. On Linux, you can use inotifywait for this. Eg:

 inotifywait -e close_write /path/to/file

If you must use lsof (maybe for portability), you could try something like:

until err_str=$(lsof /path/to/file 2>&1 >/dev/null); do
  if [ -n "$err_str" ]; then
    # lsof printed an error string, file may or may not be open
    echo "lsof: $err_str" >&2

    # tricky to decide what to do here, you may want to retry a number of times,
    # but for this example just break
    break
  fi

  # lsof returned 1 but didn't print an error string, assume the file is open
  sleep 1
done

if [ -z "$err_str" ]; then
  # file has been closed, move it
  mv /path/to/file /destination/path
fi

Update

As noted by @JohnWHSmith below, the safest design would always use an lsof loop as above as it is possible that more than one process would have the file open for writing (an example case may be a poorly written indexing daemon that opens files with the read/write flag when it should really be read only). inotifywait can still be used instead of sleep though, just replace the sleep line with inotifywait -e close /path/to/file.

6
  • Thanks, I wasn't aware of inotify. Unfortunately, it's not installed on my box but I'm sure I'll find a package somewhere. See my edit for a reason why I need the file to be closed: it's a dataset and it has to be complete before processing it further. – Peter Kovac Dec 29 '14 at 10:11
  • 1
    Another side note: while inotifywait will prevent the script from "polling" two often, the OP still needs to check lsof in a loop: if the file is opened twice, closing once could trigger the inotify event, even though the file isn't ready to be manipulated (for instance, in your last snippet of code, your sleep call could be replaced with inotifywait). – John WH Smith Dec 29 '14 at 10:14
  • @John a close_write should be ok since only one process can have the file open for writing at a time. It does assume that another one won't open it straight after it is closed, but then the same issue exists with lsof polling. – Graeme Dec 29 '14 at 10:20
  • 1
    @Graeme While this could be true by design in the OP's case, the kernel does allow a file to be opened twice for writing (in which case, CLOSE_WRITE is triggered twice). – John WH Smith Dec 29 '14 at 10:25
  • @John, updated. – Graeme Dec 29 '14 at 10:57
4

As an alternative approach, this is the perfect case for a pipe - the second process will process output from the first process as soon as it's available, rather than waiting for the full process to finish:

process1 input_file.dat | process2 > output_file.dat

Advantages:

  • Much faster in general:
    • Doesn't have to write to and read from disk (this can be avoided if you use a ramdisk).
    • Should use machine resources more completely.
  • No intermediate file to remove after finishing.
  • No complex locking necessary, as in OP.

If you have no way of directly creating a pipe but you have GNU coreutils you can use this:

tail -F -n +0 input_file.dat | process2 > output_file.dat

This will start reading the input file from the start, no matter how far the first process is through writing the file (even if it's not yet started or already finished).

3
  • Yeah, that would be the "obvious" solution. Unfortunately, the data generating process is out of my control (run by other user). – Peter Kovac Dec 29 '14 at 10:37
  • @PeterKovac That's irrelevant: cat input_file.dat | process2 output_file.dat – MariusMatutiae Dec 29 '14 at 14:14
  • @MariusMatutiae but cat and process2 could finish before process1 is finished. They wouldn't block. – cpugeniusmv Dec 29 '14 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.