Quite new to doing things on Unix, looking to make a script that does the following things in order:

  • Take main .tsv file, split into X number of files with Y lines each
  • Run each split file through a program, which outputs a new .tsv file upon completion
  • Wait until ALL split files have completed processing, then stitch output files together into one.

I know about using split and sed for splitting files, and I can't imagine getting the split files to run through a Python script is hard either, but the problem is finding out when all executions of the parallelized programs are complete, and THEN stitching their outputs together into one.

With split I know it auto-increments the names and that you can mass parallelize it as seen in this SO question, so I could figure that part out. Is there a way to check for a group of parallelized Python scripts' execution status? How could I accomplish what I'd like to do?


2 Answers 2

split -l $Y main.tsv main_part_
for part in main_part_*; do
    program $part &
echo "all done"

wait is a bash builtin: check the man page for details

  • Ended up using this, great solution, thanks!
    – Befall
    Commented Nov 4, 2014 at 22:20

As Gilles already indicated in a comment. GNU parallel is ideal for this job, as it has built-in facilities for splitting and keeping the order of split segments for rejoining. Normally it splits on lines, but you can specify specific record start and end for this as well as order it to repeat a header for all split jobs (if the .tsv are tab separated value files, this could be the column header and make it easier to write your processing program). I have used this to parallelize execution of xz.

This work most easy if your processing program is a filter, taking input from stdin and writing output to stdout. In the following assume your python program is called xyz

The basic invocation would be

cat input.tsv | parallel --pipe -k xyz > output.tsv

The --pipe option makes parallel interpret the input as data to split and to be sent to the program to invoke (it has other modes) and the -k keeps the output ordered.

The manual section on --pipe goes into detail about chunk size; record start and end (which by default are copied but can be suppressed); and repeated header.

If your xyz program needs commandline options, you can just specify them on the commandline before the output redirection (> ..)

If you have a recent version (you should anyway) you can use the more efficient --pipepart option, assuming that the input is a file (i.e. seekable) and record and linecounting is not used:

 parallel -a input.tsv --pipepart -k xyz > output.tsv
  • 1
    If input.tsv is big (1 GB or more) and xyz is fast (10 MB/s or faster), you should look at --pipepart, too. Otherwise GNU Parallel may be the bottleneck.
    – Ole Tange
    Commented Oct 30, 2014 at 8:06
  • @OleTange Cool, had not seen that yet. Hope you realise you did not put that in the tutorial yet? For those not familiar with it, it is indicated as beta in the man page of parallel version 20141022)
    – Anthon
    Commented Oct 30, 2014 at 8:06
  • I'm trying to use this since yesterday, but my network admin must've gotten the installation wrong. It runs "man parallel" but running something like "parallel echo ::: A B C" should print those three but does nothing. And the second example you gave fails to run also. I'll let you know if I get a solution actually working!
    – Befall
    Commented Nov 4, 2014 at 21:10
  • @Befall Please do a parallel --version it should say something like GNU Parallel 20141022. If not you might have the parallel from moreutils
    – Anthon
    Commented Nov 4, 2014 at 21:19
  • parallel --version gives an invalid option -- '-' response, what the hell. I asked to install from moreutils, sigh.
    – Befall
    Commented Nov 4, 2014 at 21:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .