Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I'm having two separate directories. The user loads a file into the first. Theres a cronjob running in the background which copies the files every 5 minutes over to the second directory.
What happens if the user has not completed his upload and the cronjob copies the files? Note that the two directories are owned by different users, the cronjob is performed as root.
cp does not know about opened files. So if first user uploads big file and cronjob (or any other process) starts copying this file, it will only copy as much as was already written. You can think about this in this way - cp makes copy of what is currently on the disk, no matter if the file is complete. Otherwise, you could not copy log files for example.
fuser+cp. Such copy would really be very unreliable. It won't copy any file that is opened in text editor for example.
Oct 10, 2012 at 13:14
lsof? The output of that is meant to be easy to process. You could filter the files being opened (say, by an instance of cp) for writing.
fuser of course) as this tool may not show all the files.
Oct 11, 2012 at 12:02
cp doesn't know what other programs may have the files open. There's no magic in cp. The design of unix purposefully avoids putting any kind of locks on files unless there's a compelling reason (compelling meaning the kernel needs it). On this topic, see Does redirecting output to a file apply a lock on the file?
Such situations, where a file is produced by a producer and, once complete, consumed by a consumer, are common. The usual way to handle this is to have the producer write a temporary file that the consumer will not look for, then once the producer is finished move the file into a place where the consumer will find it. Moving a file (on the same filesystem) is an atomic operation: at some point, for the consumer, the file changes from not being there to being there.
So arrange for your upload job to move the files to a different directory when it's finished doing the upload. Point the cron job at this different directory.
It seems like you want to do a dir sync job.
Because the -u, --update option of cp
copy only when the SOURCE file is newer than the destination file or when the destination file is missing
So you can add a cronjob such as cp -auv SOURCEDIR/* DESTDIR which will copy those files whose modification time have changed. That means DESTDIR will eventually get the complete copy while the uploading has finished.
rsync can do the same job. e.g., rsync -av SOURCEDIR/ DESTDIR.
Although -a option is applied, some specified attributes(e.g., ownership) can only be preserved by super-user.
See man cp, man rsync for details.
cpwill not wait until the file is completely uploaded. As we expect that the network transfer rate is lower than just copying the file from one location to another inside the same host then at some pointcpwill reach the current end-of-file and will stop copying. The solution to your problem may by simple: first the user uploads the file with some specially mangled file name (for example prepended with.(dot character). When the transfer is done user renames it to the original name. Then the cron job looks only for the files that are not starting with..