How to merge data from multiple background jobs back to a single data stream in bash

Question

I've run into a couple of similar situations where I can break a single-core bound task up into multiple parts and run each part as separate job in bash to parallelize it, but I struggle with collating the returned data back to a single data stream. My naive method so far has to create a temporary folder, track the PID's, have each thread write to a file with its pid, then once all jobs complete read all pids and merge them into a single file in order of PID's spawned. Is there a better way to handle these kind of multiple-in-one-out situations using bash/shell tools?

Answer 1

My naive method so far has to create a temporary folder, track the PID's, have each thread write to a file with its pid, then once all jobs complete read all pids and merge them into a single file in order of PID's spawned.

This is almost exactly what GNU Parallel does.

parallel do_stuff ::: job1 job2 job3 ... jobn > output

There are some added benefits:

• The temporary files are automatically removed, so there is no cleanup - even if you kill GNU Parallel.
• You only need temporary space for the currently running jobs: The temporary space for completed jobs is freed when the job is done.
• If you want output in the same order as the input use --keep-order.
• If you want output mixed line-by-line from the different jobs, use --line-buffer.

GNU Parallel has quite a few features for splitting up a task into smaller jobs. Maybe you can even use one of those to generate the smaller jobs?

Answer 2

What you propose seems pretty sensible, as it avoids having to think about how to merge the data before it's complete. So, honestly, not a bad approach!

Another common solution is to have a central program that collects the data, understands data "piece" semantics/boundaries and merges things as you go.

How to implement that depends a lot on the kind of data you're generating! This can be as simple as a really minimal program just reading messages e.g. out of UNIX or UDP or TCP sockets (but you might need to then think about having a serialization format where it's possible to know at which point some data point is complete), one socket per worker. Or just run a small relational database server (PostgreSQL?). Or you use e.g. ZeroMQ sockets to have multiple publishers and the central merger as subscriber to these, with the added benefit that this instantly also works over network. Or you use a database for time-series data. Or your data looks more like logging messages, so you implement workers that log the results via syslog or sd_journal_print, and use journald's logging namespaces to put all these log messages into a single file. Or...

In the end, your options really are:

1. write many files, merge after the fact (here you're using the fact that a filesystem allows for concurrent write access by different workers to different files, with no problem).
2. use some kind of pipe/socket/inter-process communication method to send messages to a central process (here you're using the fact that you know how your data is structured and can do the merging live).

How you actually do that 100% depends on your data structure, amount and the way you want to merge.