1

I am looking to optimize this find command. We have to move a bunch (sometimes almost a million) files to a directory for further processing. We are looking for a way to increase the speed of this command as it can take hours to run with a large amount of files. The problem is we need the files to be at least 1 minute old before moving them as we don't want to move ones that could still be being written to.

find path -name *.cdr.xml -mmin +1 -exec mv {} /path

Edit: This command is being ran as a part of a python script and we MUST wait for all the files to be found and moved prior to continuing on with the python script.

All the file names are UUID's.

0

2 Answers 2

1

One performance improvement could be achieved by using the + ending with the -exec action of find which will cause -exec to be forked only once.

While in case of ;, the -exec will be forked each time for a new argument.

So you can use :

find path -type f -name '*.cdr.xml' -mmin +1 -exec mv -t /path {} +

If there is a chance of duplicate file names :

find path -type f -name '*.cdr.xml' -mmin +1 -exec mv -it /path {} +

Or

find path -type f -name '*.cdr.xml' -mmin +1 -exec mv --backup=numbered -t /path {} +

--backup=numbered of mv will result in a numbered backup of files instead of overwriting in case of same file names in the destination.

8
  • The file names are UUID's and the processing happens immediately after this command happens, we are just moving them as more come in constantly. Aug 28, 2015 at 22:31
  • @electrometro That means very little change of duplicate names, check my edits..
    – heemayl
    Aug 28, 2015 at 22:35
  • The other concern is we need the find command to be a blocking command as this is being ran as a part of a python script that cannot continue until all the files are found and moved. Does the + at the end of exec stop the blocking? Aug 28, 2015 at 22:36
  • Yes the + will block until all the files are moved, but you might get to another problem with this method because it concatenates all the results into a single command, and the shell has a limit. If you pass millions of files this way it surely will fail.
    – Dalvenjia
    Aug 28, 2015 at 23:45
  • 1
    @Dalvenjia, heemayl: I don't think so.  I'm pretty sure that find … -exec … + will do as many execs as it needs to stay within the relevant limits.  And, BTW, unless you do -exec sh -c "…" +, the shell has nothing to do with it — find executes the requested program (e.g., mv) directly. Aug 29, 2015 at 5:37
0

You can take a look at Watchdog (https://github.com/gorakhargosh/watchdog/) to keep all your logic in python, you could watch the directory and move each file after a minute without modification.

The other solution is to use rsync to transfer all files from find output instead of constructing with -exec.

A little explanation:

find ... -exec ... \; -> This will execute the command for each result, this add a lot of overhead and is very slow.

find ... -exec ... + -> This will concatenate all the results and execute the command just once, this is very fast but you may reach the shell's character limit for a command (specially with millions of files).

Now with rsync:

find path -name *.cdr.xml -mmin +1 -print0 | rsync --remove-source-files -0 --files-from=- ${DESTINATION}

Explained:

find ... -print0 | -> This will pass all the results NULL separated to the pipe to be processed by the rsync on the other side.

--remove-source-files -> This is self explanatory, just like mv.

--files-from=- -> This will read stdin from the pipe

-0 -> This will read NULL terminated list instead of lines from the pipe.

Why NULL?... To speed up a little bit, NULL takes less bytes than a new line, take this to a million and you save a bunch of bytes.

2
  • 3
    No, on Unix & Linux, a newline is one byte.  The reason to use -print0 and -0 is that filenames can contain newlines, so a line-by-line list of filenames is inherently ambiguous.  Filenames cannot contain NULLs, so a NULL-separated list of filenames is unambiguous, even in the presence of filenames containing newlines. Aug 29, 2015 at 5:45
  • Waiting for another big batch to try this one out. Might be a few days before we have one. Sep 1, 2015 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.