Suppose if i have a situation in which some files need to be copied and it takes long time so i would have to go for parallel processing of the file copy. for example it may look like this

for i in ipaddresslist

cp x y &  //running in back ground or some other process in background


wait  //will cause till all the files are copied or whatever process if all are finished

now i have used wait so that all the backgroud processes are completed but there are cases in it like

1) some file copy may happen earlier so if i have to do some processing on these files i have to wait till all the files are copied.

2) if the copy process (or any other program running in back ground) writes to a log file the log file could be garbled with each background process trying to write to the file at the same time.

is there any workaround for such things, 1)i mean if i can know that this process has finished then i could start doing the rest of the processing for that particular instance of the process(say file copy) if its completed. also, moreover writing to log files could be in orderly fashion.

please make suggestions

  • how do people handle such situations if they occur – munish Mar 9 '13 at 10:26

If some job needs to be started after some file has been copied, just make it part of the background job:

(cp this /there && start job that needs this in /there) &
(cp that /here && start job that needs that in /here) &

(the last & is not necessary).

Now for more complex dependencies, you could use GNU make -j.

make -j2 -f /dev/fd/3 3<< 'EOF'
all: j1 j2 j3
.PHONY: cp1 cp2 cp3 j1 j2 j3 all

    cp this /there

    cp that /here

    cp this /here

j1: cp1
    start job that needs this in /there

j2: cp2
    start job that needs that in /here

j3: cp1 cp3
    start job that needs this in /here and /there

-j2 would run up to 2 jobs at any given time, and dependencies would be respected.

Now to avoid garbling of log files you have two main options

  1. don't interleave them, that is append the content of each job one after the other.
  2. try to ensure they interleave nicely, possibly tagging each line of each job to make it easier to see what job which line belongs to.

For 1, the easiest is to store each job output in a separate file and to merge them afterwards:

(cp this /there && start job that needs this in /there) > j1.log 2>&1 &
(cp that /here && start job that needs that in /here) > j2.log 2>&1 &
cat j1.log j2.log > jobs.log

Another option is to use pipes to gather the output of each job and have cat merge them. Shell process substitution as available in ksh, zsh or bash can help us with that and even take care of the backgrounding:

j1() { cp this /there && start job that needs this in /there; }
j2() { cp that /here && start job that needs that in /here; }
cat <(j1 2>&1) <(j2 2>&1) > jobs.log

j1, j2 and cat will be started concurrently and inter connected with pipes.

However note that cat will only start reading from the second pipe (that is written to by j2) after j1 has finished. That means that if j2 writes more logging than the size of the pipe (for instance, on Linux, typically 64kiB) j2 will be blocked until j1 finishes.

That can be avoided by using sponge from moreutils, like:

cat <(j1 2>&1) <(j2 2>&1 | sponge) > jobs.log

Though that would mean all the output of j2 would be stored in memory, and cat will only start writing the output of j2 in jobs.log after j2 has finished, in which case using pv -qB 100M for instance may be preferable:

cat <(j1 2>&1) <(j2 2>&1 | pv -qB 100M) > jobs.log

That way j2 would only get paused (if j1 has not finished yet) after 100M (plus two pipe contents) of logs have been output, and pv wouldn't wait for j2 to finish before outputting to stdout.

Note that for all the above, you need to beware that once you redirect the output of most commands to a file or pipe (anything but a tty), the behavior is affected. The commands, or rather the stdio API of the libc they call (printf, fputs, fwrite...) detects that the output is not going to a terminal and perform an optimisation by outputting in big chunks (several kilo-bytes), while they don't do that for standard error. That means the order of the output and error messages will be affected. If that's an issue, on GNU systems or FreeBSD (at least) and for dynamically linked commands, you can use the stdbuf command:

stdbuf -oL j1 > j1.log 2>&1

instead of:

j1 > j1.log 2>&1

to ensure that the stdio output is line-buffered (each line of output will be written separately as soon as they are complete).

For option 2, writes to a pipe of less than PIPE_BUF bytes which on Linux is 4096 bytes so much larger than your average line of log, are guaranteed to be atomic, that is if two processes write to the same pipe at the same time, their 2 writes are guaranteed not to be intertwined. There's no such guarantee on regular files but I seriously doubt that 2 writes of less than 4kiB could end up intertwined on any OS or filesystem.

So if it weren't for that buffering described above and if log lines were output individually as a whole and separately, you'd have a guarantee that the lines of the output wouldn't have a piece of line of this job and a piece of line of that other job.

However, nothing prevents a command to do a flush in between two parts of a line being written (like printf("foo"); fflush(stdout); printf("bar\n");) and there's no buffering on stderr.

Another problem is that once the lines of all the jobs are interleaved, it's going to be hard to tell which line is for which job.

You can solve both problems by doing something like:

tag() { stdbuf -oL sed "s%^%$1: %"; }
  j1 2>&1 | tag j1 &
  j2 2>&1 | tag j2
} | cat > jobs.log

(note that we don't need the wait (and it wouldn't work anyway in most shells), because cat will not finish until there's nobody writing to the pipe anymore, so not until j1 and j2 have terminated).

Above we use | cat to have pipe with its atomicity guarantee. We pipe the output of each command to a command that tags every line with the job name. j1 and j2 can write their output however they want, sed (because of stdbuf -oL) will output the lines (with the tag prefix) as a whole and separately, which will guarantee the output not to be mangled.

The same note as above still applies: we're not applying stdbuf -oL to the commands in j1 and j2 so they will most likely buffer their output which may therefore be written long after it has been produced. That is even worse than in the previous case, because if you see:

j1: doing A
j1: doing B
j2: doing C

That does mean that j1 did A before doing B, but not that it did any of them before j2 doing C. So once again, you may need to apply stdbuf -oL to more commands if it's an issue.

Note that you can't apply stdbuf to shell functions like j1 or j2 above, but at least with GNU and FreeBSD stdbuf, you can use this to set stdbuf globally or on a per-subshell basis:

stdbuf_LD_PRELOAD=$(stdbuf sh -c 'export -p LD_PRELOAD')
line_buffered_output() {
  eval "$stdbuf_LD_PRELOAD"
  export _STDBUF_O=L
j1() (line_buffered_output; cp this /there && start...)

If each downloaded file need procesing, do:

cp whatever file; process file &


If you are worried that log files get garbled, perhaps you should use syslog(3), perhaps through logger(1). Consider using nohup(1) if somebody could log out (and kill background running processes indavertently).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.