12

I download multiple files to a folder downloading via HTTPie. A bash script aims to process downloaded files, and I tried to copy the downloaded files to another folder as

find /folder/downloading -type f -exec mv '{}' /folder/downloaded \;

but this also copies the files, which have not been finished yet. I tried to limit the transfer to older files by adding -mmin +5 to the command. What is the efficient command to leave the files are being written and transfer downloaded files only?

  • If copying to same filesystem, and you set downloader to not rename (download to %.part then rename to %). Then if the downloader is well behaved (does nothing else odd), then you should be able to rename (mv) files. – ctrl-alt-delor May 29 '17 at 15:24
  • 1
    What kind of processing are you trying to do? There's most likely a much simpler option available, like using a pipeline. – gardenhead May 29 '17 at 19:35
11

Not very efficient, but you could do:

find /folder/downloading -type f -exec sh -c '
  for file do
    lsof -F a "$file" | grep -q w || mv "$file" /folder/downloaded
  done' sh {} +

That is check that the file is not listed with a write access mode in the list of open files before moving.

The psmisc implementation of fuser as typically found on Linux-based operating systems has a -w function (to check for files open for writing) but unfortunately it only work with -k to kill the corresponding processes. However, it seems you can still use it by using the pseudo-signal 0 that does nothing:

find /folder/downloading -type f -exec sh -c '
  for file do
    fuser -s -w -k -0 "$file"  || mv "$file" /folder/downloaded
  done' sh {} +

Remove the -s (or even replace it with -v) if you want to see what process(es) is(are) preventing the move.

Note that if you're not running those commands as super-user, you will only get information about your processes. If the processes downloading the files are running as a different user, they will remain undetected.

Also note that unless you're moving the files to a different file system, moving the files will not prevent whatever process is currently writing to the file from finishing writing to it.

However, depending of what they've been designed to do afterwards, they might be confused if after they finish writing, the file is no longer there (for instance if they want to change some attributes of the file after downloading it and do so not via the file descriptor (like chmod() vs fchmod(), or utimes() which cannot be done via a file descriptor)).

  • Shell question: I think I understand your code with one exception. The -exec is running sh with one command (-c '...') and the {} + tells it to put more than one result into that command at a time. What I don't understand is why there's another sh after the command. What am I missing? – Joe Jun 3 '17 at 9:39
  • 2
    @Joe, that's what goes into the inline-script's $0. That is, it gives a name to that inline script. That name may be used in error messages like <name>: fuser: command not found for instance, so sh is generally a good choice in this case to make it clear what is reporting that error message (that's also what it would be if we didn't give any argument at all after the inline-script). – Stéphane Chazelas Jun 3 '17 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.