Assume I have a landing folder /opt/landing where an external system will place multiple files all over the day. Some files are small in size and some are huge in size. We want to write a piece of script to copy files that are fully copied from landing folder to available folder /opt/available. We do not want to copy files that are in transit.

How can we achieve this using a UNIX script code.

  • this is impossible without knowing how your external system places the files there. In general "finished" is something that is an interpretation of the contents of a file, not of the file in itself. So, the question is: how do you know a file is finished? For example, if the external system uses some network protocol that supports resuming, how do you know you're looking at a finished transfer and not something that will be finished at a later time? Sep 9, 2022 at 10:48
  • If the files in transit are written with some temporary name (a random hidden name, for example) and then atomically renamed when they are done, then it would be trivial to move the files that were completely transferred. But you don't say how the files are transferred or whether that process can be adjusted.
    – Kusalananda
    Sep 9, 2022 at 11:12

2 Answers 2


With zsh:

#! /bin/zsh -
files=( /opt/landing/**/*(D.:P) )
typeset -U inuse=( /proc/<->/fd/<->(D-.:P) )
cp ${files:|inuse} /opt/available/

Would copy the regular files in there which we're not seeing as currently opened on any process file descriptor (we're not looking for files that are mmapped in a process address space, but I don't expect that would be how the files are being landed there).

Here, we don't check whether the files are opened in read or write mode. Doing so would be more complicated as we'd need to also look in the /proc/*/fdinfo/* files or parse the output of lsof which is not straightforward.

Note that you need superuser privileges to find out about file descriptors of processes that are not yours.

Files that are being uploaded over NFS or other in-kernel network filesystem would not show up in the $inuse list. Beware files may be renamed or reopened between each operation, so it remains a brittle approach. If you could update your landing system itself to move the files after it has finished dropping them, that would be more reliable.

  • **/ glob operator that stands for any level of subdirectories.
  • <-> any sequence of decimal digits.
  • (ND.:P)/(N-.:P): glob qualifiers:
    • D: Dotglob: also consider hidden files
    • .: only regular files (not other types of files like sockets, directories, symlinks...)
    • -.: same but check the type after symlink resolution
    • :P: like realpath() obtains the canonical (symlink free) absolute path of the file.
  • typeset -U: makes the array elements Unique (removes duplicates)
  • ${files:|inuse}: array subtraction (files that are not inuse).
  • arg list too long errors if there too many files can be avoided using zargs or by using zsh's builtin cp (enabled with zmodload zsh/file).

You probably want to add a further restriction that not only should the files not be in use but that they have been copied successfully. This second criterion can best be identified by the sending client.

This is not a UNIX/Linux issue but an algorithmic one.

The usual approach to satisfy both criteria (i.e. to process only completely successfully copied files) is to transfer the files with a temporary filename suffix and have the sender rename files only on successful completion.


  • Client sends datafile.xml.tmp
  • On successful transfer, client renames datafile.xml.tmp to datafile.xml


  • Server looks for files that do not end with .tmp
  • On succesful match, server processes file in whatever manner is required
  • Server looks for files that do end with .tmp and that are not "in use" (unchanged for more than a day)
  • On successful match, server deletes such temporary files

Other approaches I've had to use over the years to handle client sending systems that cannot be correctly coded include looking for end-of-file markers in the files themselves. For example, XML files can be easily identified as complete or not by applying an XML-aware parser such as xmlstarlet, and JPEG files can be easily identified with a tool such as convert from ImageMagick.

The worst case is you have to guess that a file is complete by virtue of it having not been updated for a period of time. Unfortunately you have no way of knowing if it has been fully and completely transferred or if a network error truncated the transfer part-way through.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .