1

I am running several tools that have multi-threading capabilities. For example:

tool1 --threads=4 | tool2 --threads=4 | tool3 --threads=4

In an ideal case, I assume this should utilize 12 (4*3) threads. It's possible that the first tool is the slowest, so then the downstream tools would only utilize a fraction of the requested threads. In that case, you may want to give the first tool more threads. Is there a way to manage that?

1
  • Do you want to dynamically change the number of threads? Then your tools need an interface to do this and to report their thread usage. And you probably need a controlling process that checks the threads of all tools and adjusts their thread limits. Or do you want to use the statistics from one run to calculate the parameters for the next run? This would be the only option if all you have is a command line option --threads=number
    – Bodo
    Commented Feb 11, 2019 at 17:41

1 Answer 1

1

Based on your example, the thread management is built right into the tools you're using in your pipeline.

It doesn't matter how many threads your process is using, each process has precisely one each of standard input, output, and error. All of tool1's standard output - no matter what the value provised to the --threads argument -- will be sent to the standard input of tool2, and so forth.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .