Tag Info

New answers tagged

1

Yes, the output is mixed because of xargs -P. You're executing several subprocesses in parallel, and there is nothing to coordinate their output: they're all writing output whenever they want and it all gets mixed up. Use GNU Parallel, which is a far more powerful tool to do the same job as xargs -P. Its default is to group output from each job together. ...


1

Any time you have multiple processes outputting to the same terminal (or file) in parallel, you run the risk of their output getting interspersed (unless you arrange to do some sort of locking or use low-level system calls like write to files opened in append-only mode). As a first step, you can minimize, but not totally eliminate, the problem by having ...


6

In this specific case, you're passing -P 3 to xargs. -P max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n op‐ tion with -P; otherwise chances are that only one exec will be done. Because you're running them in ...


2

Create myFileWithCommands.txt: php index.php import file1 --offline php index.php import file2 --deleteUnused php index.php import file3 Then run parallel like this: parallel -j 3 -- < myFileWithCommands.txt


2

Well, there are two simple ways to accomplish this that I can imagine. You can either pipe the contents of the file into parallel, or you can write a shell script. If all you want to accomplish is this particular set of tasks, then a shell script might make more sense. It would be very short, sweet and to-the-point as well: #!/bin/sh parallel -j 3 -- "php ...


0

GNU tar has --append: tar -f foo.tar --append newfiles Unfortunately it reads the full tar file.


2

You can start the creation of the final tar file before all output files are created: Maybe that achieves the speed up you want. You can call tar this way: tar -cf foo.tar -T file-list file-list would be a FIFO. You need a script which detects new files in the source directory (inotifywatch) when each of these new files is finished (fuser) If a file ...


5

You can't have multiple processes adding to the same tar archive (or any other usual archive format, compressed or not). Each file is stored contiguously, and there is no way to insert data in a file, only to append or overwrite, so continuing to write to a file that isn't the last one would overwrite subsequent files. If you know the file size in advance, ...



Top 50 recent answers are included