3

I've run into a couple of similar situations where I can break a single-core bound task up into multiple parts and run each part as separate job in bash to parallelize it, but I struggle with collating the returned data back to a single data stream. My naive method so far has to create a temporary folder, track the PID's, have each thread write to a file with its pid, then once all jobs complete read all pids and merge them into a single file in order of PID's spawned. Is there a better way to handle these kind of multiple-in-one-out situations using bash/shell tools?

7
  • @Archemar I know about wait and use that in my scripts to wait till all the jobs are done to start reading in the files, but I dont see a way wait helps me get the data directly from those jobs once they complete
    – guest
    May 23 at 14:37
  • If the order is not relevant, just write the output in append mode (aka >> /common/logfile.out).
    – K-att-
    May 23 at 14:44
  • 1
    @frabjous uuuuuh, IO buffers say that is practically never a good idea, unless results are all single characters long or you can definitely 100% guarantee no buffering and result sizes small enough to fit into write buffers of your programNs read and write buffers of your shell. May 23 at 14:53
  • 1
    Yes. With append mode, the operating system can handle it. Fyi: unix.stackexchange.com/questions/346062/…
    – K-att-
    May 23 at 14:59
  • 1
    @guest in general, only single calls to write (2) on files are atomic (but if the file was opened with O_APPEND, skipping to the end of it is atomic together with that single write). So if your program writes all its output in a single call to write, fine, if it calls write multiple time, you might get intertwined output data, which might simply be broken. I'd like to add that the assumption "a program called from a shell script only uses write on stdout a single time" is a very big assumption. May 23 at 15:01

2 Answers 2

7

My naive method so far has to create a temporary folder, track the PID's, have each thread write to a file with its pid, then once all jobs complete read all pids and merge them into a single file in order of PID's spawned.

This is almost exactly what GNU Parallel does.

parallel do_stuff ::: job1 job2 job3 ... jobn > output

There are some added benefits:

  • The temporary files are automatically removed, so there is no cleanup - even if you kill GNU Parallel.
  • You only need temporary space for the currently running jobs: The temporary space for completed jobs is freed when the job is done.
  • If you want output in the same order as the input use --keep-order.
  • If you want output mixed line-by-line from the different jobs, use --line-buffer.

GNU Parallel has quite a few features for splitting up a task into smaller jobs. Maybe you can even use one of those to generate the smaller jobs?

1
  • 1
    oh nice! I think I might really need to get the hang of GNU Parallel. I just realized the documentation is way nicer than I remember it being! May 23 at 16:06
3

What you propose seems pretty sensible, as it avoids having to think about how to merge the data before it's complete. So, honestly, not a bad approach!

Another common solution is to have a central program that collects the data, understands data "piece" semantics/boundaries and merges things as you go.

How to implement that depends a lot on the kind of data you're generating! This can be as simple as a really minimal program just reading messages e.g. out of UNIX or UDP or TCP sockets (but you might need to then think about having a serialization format where it's possible to know at which point some data point is complete), one socket per worker. Or just run a small relational database server (PostgreSQL?). Or you use e.g. ZeroMQ sockets to have multiple publishers and the central merger as subscriber to these, with the added benefit that this instantly also works over network. Or you use a database for time-series data. Or your data looks more like logging messages, so you implement workers that log the results via syslog or sd_journal_print, and use journald's logging namespaces to put all these log messages into a single file. Or...

In the end, your options really are:

  1. write many files, merge after the fact (here you're using the fact that a filesystem allows for concurrent write access by different workers to different files, with no problem).
  2. use some kind of pipe/socket/inter-process communication method to send messages to a central process (here you're using the fact that you know how your data is structured and can do the merging live).

How you actually do that 100% depends on your data structure, amount and the way you want to merge.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.