Linux how to copy files that are not in use inside a folder

Question

Assume I have a landing folder /opt/landing where an external system will place multiple files all over the day. Some files are small in size and some are huge in size. We want to write a piece of script to copy files that are fully copied from landing folder to available folder /opt/available. We do not want to copy files that are in transit.

How can we achieve this using a UNIX script code.

Answer 1

With zsh:

#! /bin/zsh -
files=( /opt/landing/**/*(D.:P) )
typeset -U inuse=( /proc/<->/fd/<->(D-.:P) )
cp ${files:|inuse} /opt/available/

Would copy the regular files in there which we're not seeing as currently opened on any process file descriptor (we're not looking for files that are mmapped in a process address space, but I don't expect that would be how the files are being landed there).

Here, we don't check whether the files are opened in read or write mode. Doing so would be more complicated as we'd need to also look in the /proc/*/fdinfo/* files or parse the output of lsof which is not straightforward.

Note that you need superuser privileges to find out about file descriptors of processes that are not yours.

Files that are being uploaded over NFS or other in-kernel network filesystem would not show up in the $inuse list. Beware files may be renamed or reopened between each operation, so it remains a brittle approach. If you could update your landing system itself to move the files after it has finished dropping them, that would be more reliable.

• **/ glob operator that stands for any level of subdirectories.
• <-> any sequence of decimal digits.
• (ND.:P)/(N-.:P): glob qualifiers:
  • D: Dotglob: also consider hidden files
  • .: only regular files (not other types of files like sockets, directories, symlinks...)
  • -.: same but check the type after symlink resolution
  • :P: like realpath() obtains the canonical (symlink free) absolute path of the file.
• typeset -U: makes the array elements Unique (removes duplicates)
• ${files:|inuse}: array subtraction (files that are not inuse).
• arg list too long errors if there too many files can be avoided using zargs or by using zsh's builtin cp (enabled with zmodload zsh/file).

Answer 2

You probably want to add a further restriction that not only should the files not be in use but that they have been copied successfully. This second criterion can best be identified by the sending client.

This is not a UNIX/Linux issue but an algorithmic one.

The usual approach to satisfy both criteria (i.e. to process only completely successfully copied files) is to transfer the files with a temporary filename suffix and have the sender rename files only on successful completion.

Client

• Client sends datafile.xml.tmp
• On successful transfer, client renames datafile.xml.tmp to datafile.xml

Server

• Server looks for files that do not end with .tmp
• On succesful match, server processes file in whatever manner is required
• Server looks for files that do end with .tmp and that are not "in use" (unchanged for more than a day)
• On successful match, server deletes such temporary files

Other approaches I've had to use over the years to handle client sending systems that cannot be correctly coded include looking for end-of-file markers in the files themselves. For example, XML files can be easily identified as complete or not by applying an XML-aware parser such as xmlstarlet, and JPEG files can be easily identified with a tool such as convert from ImageMagick.

The worst case is you have to guess that a file is complete by virtue of it having not been updated for a period of time. Unfortunately you have no way of knowing if it has been fully and completely transferred or if a network error truncated the transfer part-way through.